Could someone please explain the difference between Lisp macros and, say, languages that have first-class functions? I get that a Lisp macro will be expanded into the respective code, while a function's execution is different. However, at the practical (i.e., developer's) level, are there any additional benefits?
Can, say, a Lisp macro be 'partially formed', in the sense that it can expand into some boilerplate that represents an incomplete syntax tree? (Whereas a higher-order-function is necessarily complete.) I can see that being useful, but not inasmuch a it's made out.
My favorite example for this is the lame idiom you see in Java code:
if (log.isDebugEnabled()) {
log.debug("expensive" + debug + message);
}
This is "better" than just log.debug(...) because with the latter, your expensive log message argument needs to be evaluated even if debug is disabled.
However, in a language w/ macros, you just say:
(debug (str "expensive" debug message))
and these considerations are already taken care of for you:
The point was that "debug" and "message" parts in the example were arbitrarily complex expressions which were computationally expensive to evaluate. Your wrapping of log debug would still require evaluating them to get the message string unconditionally, even when debug logging is not enabled.
Of course, with support for first-class functions, you could do something like:
func debug(produceMessage ()->string) {
if log.isDebugEnabled() {
log.debug(produceMessage());
}
}
The downside there is you can still end up allocating a closure. Whereas an expanded macro shouldn't cost anything. (A sufficiently smart compiler might be able to optimize the closure allocation.)
A sufficiently modern language (like, saaaaay, D2) could also give you a type like "closure you don't intend to escape" (let's call this a "scoped closure", or if you will, "scope string delegate()"), and eschew allocation entirely without requiring optimization.
For completeness of the argument, this particular problem is solved in the Java world with string formats. With the slf4j interface, that would be:
log.debug("expensive %s %s", debug, message)
The message is not actually formatted into one string unless the DEBUG trace level is enabled. Of course, you are still passing the arguments around, but with object references that's a negligable difference.
I still appreciate the solid example of a problem macros are good at solving, though. Two ways around one problem.
It's not solved, because method arguments are evaluated eagerly. It means that message argument may not be an expensive expression, because it will be calculated independently whether debug is enabled or not. In case of macros arguments of this method could be evaluated lazily.
Calling log.debug() with and argument of `false` is a no-op? That sounds like someone bending the language to fit an idiom, because it doesn't sound like a sane API except that it enables this use case.
It is insane in an eager-evaluated calling context. You're not wrong. But in Lisp it's not insane at all to let a macro consume a parameter and yield a no-op. Think of how this plays with a JIT and having code that can dynamically switch between dev/QA/production behavior and performance profiles, just for one example.
That does something similar, sure, but what it in fact is is a macro. As snikeris was saying, you couldn't write anything quite like that #define in a macroless language like Java.
One difference is flow control. When you call a function, all the arguments are evaluated before being passed into the function. If you want to delay evaluation, you have to wrap the argument values in a function. When you call a macro, the text forms get passed with no evaluation.
Say Clojure forgot to ship with the boolean "or". "or" should evaluate its arguments one at a time (to allow for short circuiting) and return the first non-false value. You could do this with functions, but you have the source-level overhead of manually wrapping everything, and performance overhead of defining and passing an anonymous function for each argument. With a macro, at compile time you just translate the "or" macro into a simpler form that uses "if". This is how Clojure actually implements "or":
(defmacro or
"Evaluates exprs one at a time, from left to right. If a form
returns a logical true value, or returns that value and doesn't
evaluate any of the other expressions, otherwise it returns the
value of the last expression. (or) returns nil."
{:added "1.0"}
([] nil)
([x] x)
([x & next]
`(let [or# ~x]
(if or# or# (or ~@next)))))
You should have a look at Kernel, vau-calculus and F-expressions. Kernel merges the notions of first-order functions and macros by having lexically scoped definition of Fexprs which takes their argument by name and also receives the call-site environment so that you can eval the argument in it if needed.
I find it a very elegant way of having everything clean, "lambda" does not even have to be a primitive anymore.
Look at the uses for Template Haskell, for example.
That's basically Haskell's macro system. And it's used quite a lot, though it's kind of arcane.
If you want to create an abstraction that defines one or several data types, you'll think hard about whether you can use some kind of type-level programming instead—but if that's not possible, or not convenient, you can use TH macros.
For example, the `lens` package defines TH macros for creating special kinds of accessors that are tedious to write by hand.
Which things cannot be expressed without macros? Macros run at compile time, and just output normal code in the language.
I think being able to generate types is a very useful and important use of macros. In fact, (depending on whether or not your macros can have side effects) you could use macros to implement something like F#'s type providers.
The majority of macros I write could be represented with HOF and lexically-closed lambdas. That adds significant extra syntax when you use them though.
A minor advantage is that a macro will be expanded in-line; a Sufficiently Smart Compiler could transform the HOF version into something equivalent, so it's not strictly an advantage (except to compiler implementers I suppose).
I think that e.g. generalized references (i.e. the common lisp setf macro) are not possible with HOF, but HOF is more common in pure languages where assignment is eschewed anyway.
Mark Jason Dominus (author of Higher Order Perl) had some good comments on lisp macros as well:
One thing to note is that you're using the macro or lambda to delay evaluation. In a lazy-by-default language, that's unnecessary (which is a part of why macros are less useful in Haskell).
There's a class of things that don't require macros in Haskell, but I don't think that means macros are less useful in Haskell. There are plenty of things you might want them for, like generating new definitions.
Macros can provide a lot of syntactic convenience over those first-class functions, especially with heavily nested structures. For example, I can replace this monadic parser definition...
This requirement arose in Haskell for its Parsec before it became a part of the syntax, then again later with the Arrow library which later also added it to the syntax. Whenever new abstractions are discovered/created, a macroing facility, whether for lisp-like syntax or some other, helps makes all the nested functions more readable.
A higher order function doesn't serve the same purpose as a macro. A higher order function is meant to be applied, called, composed etc. A lisp macro is a different type of abstraction. For example, many people think that macros are just hiding lambda's of higher order functions. This is wrong. A macro abstracts over implementation details of a construct to make it read naturally. For example, you can write a function that opens and then closes a file like so...
with-open-file(filename, lambda file: do stuff with file)
but with a lisp macro, you only have to write
with-open-file (filename):
do stuff with file
The point of an abstraction is so you don't have to think about the implementation. Written like the latter, with-open-file is simply more natural to write this way. You only have to think, "oh, its a construct that opens a file then closes it after the body is done", rather than "oh, its a higher order function that i have to pass another function into that takes the file as an argument..." etc.
When you write with-open-file as a macro, it could be implemented as a higher order function, or it could be implemented as a low level set of GOTO statements. It doesn't matter. The macro abstracts away the low level detail, just providing the most natural way for you to use the construct.
It might not seem like the macro is doing much in that particular example, but a construct that defines a class (like defclass) is something you can write as a macro, which can expand into functions that do the actual defining. You could write a class defining construct as a higher order function, but then you'd have to constantly worry about how your construct was implemented as a function. instead of just writing something natural like
defclass tiger (animal):
age init-value: 0
name type: string
which you could write if you implemented defclass as a macro.
otherwise,
you'd have to do something crazy stupid like
how could a higher order function possibly implement that without making people using it tear their hair out?
They have to bend to the implementation, not make the abstraction bend to what's natural.
Your point seems to be that macros allow for a slightly more natural syntax for certain things, but I can do pretty much the same thing in a language with a natural HOF syntax (Ruby):
with_open_file filename do |f|
do stuff with file
end
And for your second example:
# our 'macro' function
def defclass(name, parent, &blk)
k = Class.new(parent)
k.instance_eval(&blk)
Kernel.const_set(name, k)
end
defclass :Tiger, Animal do
attr_accessor :age, :name
end
Yes, macros allow for a superset of what you can reasonably accomplish with higher order functions, but I haven't yet seen a simple practical example of where the added power is useful. I'm sure it's nice to have, but there's something to be said about a language which generally gets you 95% of the way there using a simple set of built-in operations.
Sure, ruby has nice syntax for lambda, but the entire point is not even having to worry about what goes into blocks and what not. Syntax is a bad excuse for abstraction. Lisp has higher order functions too, but a deceptively short (or "natural") syntax can easily sweep ugly semantics under the rug. I know no one would ever write your defclass hof anyways because of the extreme runtime overhead that causes. If you wanted to reimplement your defclass macro more efficiently, I don't see how you would be able to do so while not breaking all of a users code relying on that function. The point of a macro is so the implementation detail is hidden, and the interface doesn't have to change when you change the implementation. Lisp has higher order functions too. Hell, that's where ruby got it from. Higher functions have their uses, but they are not for abstracting patterns in code like macros are. They are for abstracting procedures in code on arguments received at runtime. The difference is macro expansion happens lexically, at compile time, while functions are part of the program at runtime.
If you haven't seen a practical example of what it's useful for, that's akin to the attitude of a C programmer not understanding the usefulness of higher order functions. They say, I get 95 percent of the way their with good old functions and function pointers. We would find that absurd, just as how I find the claim that a 'simple practical example of where added power of a macro is useful does not exist' is absurd
For example, see 'A unit testing framework' in 'practical common lisp' available online by Peter seibel. I can't possibly imagine how you'd be able to create a unit testing framework abstraction in Ruby as nice or efficient as the one presented with higher order functions in 26 lines of code. But it'd be cool if anyone could prove otherwise.
Notice that in your Ruby code you have to use quoted symbols like :Tiger and :age and :name, because you cannot extend Ruby's syntax with your own. Ruby has good metaprogramming facilities, but it's no substitute for a real macro system.
Semantically, I'm writing in common lisp. If you mean the syntax, that's an ad hoc thing that comes up when writing in an HTML text box and emacs isnt here. But the syntax is shallow and unimportant compared to the abstraction presented. (it's just normal lisp syntax with implied parentheses anyways, so it's a bit ambiguous)
Well, typing parens in a HTML textbox can be pretty tedious, so I understand the desire for pseudocode.
Dylan has an interesting hygienic macro system that's similar to Scheme's. I don't think Dylan allows for arbitrary code-generating macro procedures, but it's possible in principle; see [1].
It's true, as others have explained, that for many of the most common uses of macros, you can get the same effect with a higher-order function.
But since macros operate at the syntax level, they can do things functions can't do. For example, they can generate and manipulate declarations. Say you're working with abstract syntax trees (assembly trees in a CAD app might be another example). These trees are built from nodes, where each node is of some class corresponding to a syntactic construct: if-statement, addition-expression, etc. etc. There's some functionality you want to have on every node class; a common example is a "children" method that gathers up all the node's child slots into a set and returns it. It is very convenient to have a 'define-node-class' macro that automatically generates the 'children' method, so that when you add a child slot, the method is updated automatically; there's no need for manual effort to keep them in sync.
In this case, the macro is expanding to multiple top-level declarations: the class declaration along with the method declaration (probably, in practice, several methods). Higher-order functions don't begin to let you do stuff like this.
Because most node classes have their own slots. Consider this example:
class Node {...}
class Expression extends Node {...}
class Addition extends Expression {
Expression left, Expression right;
Set<Node> children() {
return [a set containing 'left' and 'right'];
}
}
There's no way to write a single method 'children' on Node that will work for all its subclasses, because the method on Node can't access the subclasses' slots -- unless you use reflection, which is ugly and slow.
The sheer number of answers you've got should tell you that indeed there is something about those macros. But you need to 'get' them yourself to appreciate them.
Each answer exposes one or more facets of macros. Indeed, there is not just 'one' thing that make them worthwhile.
Ah, but that is not the case in some languages in which arguments are evaluated lazily. Usually the iron-man version of this question is: "if I have a non-strictly evaluated language with higher order functions, do I still need macros?"
A part of the answer is: you probably don't need the kinds of macros which cover up machine-generated lambdas, which simulate non-strict evaluation in strictly evaluated Lisp programs! The compiler for your language already has these "macros" in its compiler.
The argument why "you always need macros no matter what else you have" is that your language has "hard-coded macros": the grammar rules in a compiler, which match patterns and transform bits and pieces to produce AST fragments. If you don't have macros, then that set of "hard-coded macros" is all you have.
So, even in a functional language with nonstrict evaluation, you're using macros. It's hard to make a convincing argument that they provide all the expressivity you would conceivably ever need. (And their existence and use defeats any argument that you don't need macros at all).
Which definition of "macro" are you using, if you equate lazy evaluation with a kind of "hardcoded macro"?
More importantly, "it's hard to make a convincing argument that [these hardcoded macros] provide all the expressivity you would conceivably ever need" doesn't convince me. A better argument would be to produce a compelling example where these "macros" are not enough. And by compelling, I mean something that cannot be elegantly produced in a non-Lisp language.
> Which definition of "macro" are you using, if you equate lazy evaluation with a kind of "hardcoded macro"?
That definition of "macro" equivalent with "phrase structure rule in your functional language's compiler" which takes the input structure and generates whatever code brings about the lazy semantics (which is not inherent in the x86 instruction set or what have you).
> example where these "macros" are not enough
An example is any instance of language extension where, say, the maintainers of the compiler for a functional language have to ship a new compiler to the users to get them to use the new feature.
Functional languages with lazy evaluation are not finished, right? They are developed actively.
> An example is any instance of language extension where, say, the maintainers of the compiler for a functional language have to ship a new compiler to the users to get them to use the new feature.
> Functional languages with lazy evaluation are not finished, right? They are developed actively.
Agreed, they are actively developed.
Correct me if I'm wrong, but you seem to be saying "many interesting features can be implemented in a Lisp language with a macro, therefore Lisp programmers don't need to wait for a new release of their programming language when they want these features".
I'm not convinced this is the case, or rather, that this is such a relevant case. Aren't Lisps actively developed too? Why is there such a multitude of Lisp implementations? Is any relevant real-world feature truly implementable with Lisp macros? Why is that more convenient than implementing them with functions in languages with lazy evaluation?
Maybe I'm falling prey to the Blub paradox. I'm only passingly familiar with Racket, thanks to a course in Coursera, where they introduce macros and why they are so powerful in Lisp. But I still don't see the compelling "killer example"...
> Correct me if I'm wrong, but you seem to be saying "many interesting features can be implemented in a Lisp language with a macro, therefore Lisp programmers don't need to wait for a new release of their programming language when they want these features".
That is correct. However, macros have to translate to something. From time to time you need an upgraded something for some macros to be feasible.
E.g. it's hard to "macro your way" into having first-class continuations, if they aren't in the substrate.
That is to say, without the macro treating all of its arguments as a self-contained language.
You usually want the code which is in the macro forms to smoothly interoperate with outside code, such as make lexical references to surrounding bindings.
This is the fuzzy limit. Macros have to write stuff in some language, which is no longer macro-expandable. That language has to be reasonably expressive in its semantics for what the macros want to do.
So you think of a feature you'd like in your language. Let's consider the process you'd have to go through to use that feature.
In most languages, you have to write the maintainers about the feature. You then have to convince them that the idea is good -- and this is by no means guaranteed; if they think your idea is bad, you're out of luck, and can never use the feature you'd like. Then someone has to implement it. Then you have to wait for a release containing the feature. Then you have to test your code with the new version. Then you have to upgrade all your deploys, development environments, and testing machines to the new version. Then you can use the new feature.
In a Lisp with macros, you have to implement the feature. Then you can use it.
This is why macros are useful. You get to modify the language you're using, but still cut out the entire loop of the maintainers of that language.
Yes, I understand that argument, but I still find it unconvincing.
Some features you just can't implement with macros, you need a change in the "substrate" (see kazinator's answer below). For the rest, I simply don't see how they are language-level features. They are just things you need for your project, in which case, why can't you simply implement them as a library?
Even if you ignore the above, there's probably a good reason why the language designers don't want to approve your language-level feature. Yes, sometimes it's simply red tape or politics, but it can also be that you -- the applications programmer -- simply aren't well-versed in language design and can't think past your particular use case :) This wouldn't mean the feature is worthless (after all, you need it!) but maybe it's not meant to be a language-level feature, but instead... a library function, which you can write in most general purpose languages.
But this was only added in 2004!^2 So for almost 10 years, you had to manually iterate over stuff. How would you implement this as a library? Well, you could write a function that lets you write:
Would this work in a lambda? Well, if the language you're using has real closures, yes -- but does it? Do you know offhand? With a macro, your code will work.
> Even if you ignore the above, there's probably a good reason why the language designers don't want to approve your language-level feature. ... it can also be that you -- the applications programmer -- simply aren't well-versed in language design and can't think past your particular use case :) This wouldn't mean the feature is worthless (after all, you need it!)
That seems like evidence for my point -- macros let you build the language up for your own use case, not anyone else's. Without macros, your choices are "either everyone can use it, or no one can use it". Macros let you have a choice of "well, I can use it, even if no one else wants it, if I find it useful."
Macros can be viewed as libraries that act on the language itself. There isn't a difference between language-level features and "library functions" in a language with macros.
[1] Without getting into a Turing Tarpit. We're talking about using things in easy ways, not what is technically possible but ugly and kludgy.
Agreed about your Java example. However, for the purpose of this discussion, let's assume we're talking about modern, well-designed languages without ugly kludges and with access to nice features such as lazy evaluation and real closures.
> Macros can be viewed as libraries that act on the language itself. There isn't a difference between language-level features and "library functions" in a language with macros.
I simply don't see why this is such a big deal. I need to see a real-world example (which, understandably, might be difficult to explain in a HN thread) of something that can be achieved with Lisp that is not reasonably achievable in elegant ways in other, non-Lisp modern languages. Again, let's assume we both understand the Turing Tarpit.
By the way, I don't want to sound dense. I understand some features you only "get" when you use them. What little I've seen of Lisp (Racket, actually) seemed very interesting! It's just that I can't get that enlightened moment where I see why Lisp macros are that important in the real world. This is important to me because macros are one of the key features Lispers use to try to convince other programmers Lisp is awesome. And I can see they are interesting and useful; I just fail to see why they are a such big deal that they set Lisp apart and that, for example, Paul Graham would call Lisp his "secret sauce".
Many have said that the real benefit is that you wind up turning Lisp into the language that is perfect for your domain.
When you start your project, you don't know enough to design the perfect language for your domain. You start coding in Lisp, and you begin to uncover patterns that express your domain. Eventually, you find your way towards building a small set of macros that beautifully, expressively capture your domain.
Look for where Paul Graham talks about "bottom-up" programming versus "top-down" programming, and you'll find what he has to say about this. He says you do both in Lisp. Bottom-up is "changing the language to suit your problem."
One that helped some Java friends understand is passing blocks of code, but still having it look like just writing code. Imagine instead of try/catch/finally, a transaction/commit/rollback in Java:
transaction {
// everything in here is in one transaction
} commit {
// do stuff if the commit is successful
} rollback {
// do stuff if we rollback
}
All the try's and catch's can be stuff into the macro. It can be made to nest transactions within transactions.
For all practical purposes, you can't add that to Java. You'll always have to wrap up your transactions in boilerplate.
The Lisp equivalent of what I want, the resulting code would look like:
There are languages where I could define three blocks of code and pass them:
transaction(stuff, commit-stuff, rollback-stuff)
But that separates their definitions from their implementations.
How could you write the Lisp transaction expression in another language so that it looks like it's part of the language? (serious question, CL is the only language I use capable of being that close) Maybe I could torture Ruby to come close, but it would be far more difficult than the macro I had to write for Lisp.
>For all practical purposes, you can't add that to Java. You'll always have to wrap up your transactions in boilerplate.
Aren't you wrapping the lisp code in boilerplate when doing the macro too? This appears to be the same as your other example. Java has lambda expressions (since Java 8) that could do this.
If you are wrapping it in a macro, how is it different than wrapping it in a method? In Java 8, with lambda expressions, you can write code to wrap your transaction example to get something exactly equivalent to
>transaction(stuff, commit-stuff, rollback-stuff)
and you would be keeping the definition in the implementation. I also seem to be missing why it's important to be keeping the definition and implementation together.
So there's some boilerplate that needs to happen. With macros, you write the macro to insert the boilerplate, and then you never think about it again. You don't write it, you don't read it, it's not in the way. Without macros, you have to write the boilerplate every time you write the code. You have to read the boilerplate every time you read the code.
Which do you find more readable? Notice how there's no boilerplate in the macro version.
> This appears to be the same as your other example. Java has lambda expressions (since Java 8) that could do this.
What are the odds that Java 8 has provided everything you could want in out of Java? Macros let you add things in a better way than you could otherwise get. Look back to my prior example of Java 5's expanded for. You see how useful that was? If Java had had macros, people wouldn't have had to suffer through nine years without the improved for loop.
> What are the odds that Java 8 has provided everything you could want in out of Java? Macros let you add things in a better way than you could otherwise get. Look back to my prior example of Java 5's expanded for.
Again, please understand we non-Lispers find this argument utterly unconvincing :) Java in particular is a terrible example: it's a language that for many years remained in the dark ages, and now it's finally getting modern features retrofitted into it, while at the same time attempting to keep some sort of backwards compatibility, and the whole process is very painful.
Let's all agree to stop talking about Java. Let's assume we all agree working in a language that until very recently didn't have lambdas is painful. And that requires a horrific amount of boilerplate.
Let me re-throw your question back at you: what exactly can Lisp do, which has practical implications, that a modern language with lambdas, closures and lazy evaluation cannot accomplish in elegant ways? If you mention Java again, you lose :P
You seem to be concerned by the use of "Java". It's a fine example, and one I think I've explained why very well. It also seems petulant to declare Java off-limits.
But let me try again. For _any given language_, there are things you may want that the language doesn't provide. Macros let you do that in an elegant way. The example I gave above -- based off LanceH's -- of transactions is a way where the macro-based solution is more elegant than non-macro solutions.
Here's another example. Arc, like any language, has a built-in way of setting variables. It's called `assign`, and can be used as follows:
arc> (assign a 3)
3
arc> a
3
But no one uses it. Instead, people use =. = is a macro that's provided with the language. Because it's a macro, that means that if it wasn't provided, you could write it yourself.^1
What's the benefit of = ? It lets you set values of more than just variables:
arc> (= my-table (table))
#hash()
arc> (my-table 'key) ;;look up the value
nil
arc> (= (my-table 'key) 'value)
value
arc> (my-table 'key)
value
Note that in the second prompt, we're attempting to look up the value of 'key in the hashtable my-table, and we see that there isn't one there. We then set it in the third prompt, and look it up again in the final.
This works on the concept of "places". The "place" we try to set in `(= (my-table 'key) 'value)` is the association of 'key inside my-table.
Why is this beneficial? If you know how to get a value out of a data structure, you can now set it. This code is extremely clean and understandable compared to a non-macro version. It exhibits the principle of least surprise, and it's obvious how to set other data structures in an elegant way.
You can't do this without macros.
[1] If your objection here is "but it comes with the language", you're missing the point.
I disagree your example of Java is fine. It's not petulant to declare it off-limits, just as it's not petulant to declare COBOL off-limits. We are discussing features and extensibility of finer languages than Java.
> The example I gave above -- based off LanceH's -- of transactions is a way where the macro-based solution is more elegant than non-macro solutions.
Here we disagree. In your example of transactions, you didn't shown that macro-based solutions are more elegant than non-macro based solutions; you merely showed that Java before version 8 wasn't very good (and when someone replied "but we can do better with Java 8!" you basically replied "ok, but how do you know Java 8 is enough for some other unspecified problem?"). Your answer is unconvincing, especially since non-macro-based solutions to your example in other languages, such as Scala, are equally elegant to Lisp's, because Scala has support for first-class functions and closures. Let me preempt a "but how do you know that's enough for Scala?"... I don't know. Show me why it's not enough!
Re: your second example with assign and the = macro. I admit I don't understand it yet; I'll have to think some more about it.
edit: let me go back to this assertion:
> For _any given language_, there are things you may want that the language doesn't provide. Macros let you do that in an elegant way.
I find this problematic, for two reasons. First, we've acknowledged macros cannot solve everything; after all, there are new releases and multiple implementations of Lisp languages. What someone else said: "if it's not in the 'substrate', macros can't do it". Second, that macros let you do some (admittedly cool!) things doesn't automatically show that these same things cannot be accomplished in reasonably elegant ways in other languages. One thing doesn't imply the other!
>In your example of transactions, you didn't shown that macro-based solutions are more elegant than non-macro based solutions; you merely showed that Java before version 8 wasn't very good...
I think it wasn't clear what I was referring to. I was talking about not Java, but a Lisp-style solution. Here it is with macros:
The macro-based solution -- which doesn't mention Java -- is more elegant. You don't need the lambdas if you use macros (why? Well, you don't always want to rollback, right?)
>...when someone replied "but we can do better with Java 8!" you basically replied "ok, but how do you know Java 8 is enough for some other unspecified problem?"
That's the point -- a language with macros is extensible in a way that languages without macros aren't. So unless you believe that your language happens to be perfect, having macros would make the language more powerful.
>Re: your second example with assign and the = macro. I admit I don't understand it yet; I'll have to think some more about it.
Feel free to contact me if there's anything else I can explain -- I'm probably going to forget to check this thread soon.
>...we've acknowledged macros cannot solve everything; after all, there are new releases and multiple implementations of Lisp languages.
I'm not sure why new Lisp releases show that macros are not useful. You could write anything in assembly, but other languages are still released. And design is important -- which sets of functions, and macros should be provided with a language? Does having a release of Scala that includes functions mean that there's no need for user-defined functions?
>Second, that macros let you do some (admittedly cool!) things doesn't automatically show that these same things cannot be accomplished in reasonably elegant ways in other languages. One thing doesn't imply the other!
It doesn't mean that, no. However, I don't see elegant ways to do this kind of thing in other ways. If you can show me some, I'd be interested.
The problem that I have with lisp macros is that the elegance you gain at the syntax level is effectively a tradeoff with pragmatism when other people read and use the code. Java was developed with parts of C++/C as inspiration and parts of that language were left out. Particularly, operator overloading was left out (which can be viewed as a very restricted example of modifying the language), presumably because it's not an immediate thought when viewing the code that the operator isn't doing what you expect. While it's undeniable that macros make the syntax nice to look at, the same argument that lisp becomes a new language as you write your program means that every separate codebase has a lot more reading to understand because you have to go through all the macros. New releases in most languages introduce new features (and standardize functions, fix bugs, etc), as far as I can see, new releases in lisp enforce a standard (common) base set to decrease the amount of work required in learning new codebases.
Yes, macros can make code very hard to read, if designed badly. Of course, so can functions, variable names, and program flow.
But Lisp with macros is very different from C++ with operator overloading. With C++ operator overloading, you only know if a given line has something you don't understand (that is, an overloaded operator) by looking at every other file in your project. With Lisp macros, you know that you're dealing with something new because you don't recognize the first token in the s-expression. You might not know it's a _macro_ rather than just a _function_, but you know it's something you need to investigate.
Basically, in Rumsfeld's terminology, an overloaded operator is an unknown unknown, but a Lisp macro is a known unknown. A macro's behavior may be confusing, but its existence isn't. And that's a very big difference.
Agreed, I'm still unconvinced for the same reasons. In Scala or Haskell (or any language which supports first-class functions and lazy evaluation, I guess) the "transaction" example is easily done.
DOLIST is similar to Perl's foreach or Python's for. Java added a similar kind of loop construct with the "enhanced" for loop in Java 1.5, as part of JSR-201. Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably still can't use the new feature until they're allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.
One could write a macro that allows infix notation for arithmetics:
(arithmetics 1 + 2 - 3) = (- (+ 1 2) 3)
These kind of syntactic transformations are what macros enable.
TXR Lisp is completely strictly evaluated, like many other Lisp dialects. Function argument expressions are reduced to their values, in left to right order. Then the application of the resulting values to the function takes place.
The special mlet ("magic let" or "mutual let") construct has allowed the expression which initializes s, (lcons 1 x), to refer to x.
This works because both mlet and lcons are macros. The lcons macro (rather, the code generated by the macro!) returns a lazy cons cell, without immediately evaluating its arguments 1 and x. In the case of x, this is a damn good thing because x is not yet initialized! When the lazy cons is accessed (when the list object is printed), the evaluation of x takes place. By that time, x holds the lazy cons cell, and since the variable x is in scope of the argument x in (lcons 1 x), the lazy cons is able to force, setting its CDR field back to itself, creating not a lazy list, but a circular list.
With lcons, I we can make a fibonacci function quite similarly to how you might do it in Haskell:
(defun fib2 (a b)
(lcons a (fib2 b (+ a b))))
We call this as (fib 1 1) and it gives us a lazy list.
Here, we have a marriage between a lazy data structure and a macro. Without lazy conses, the lcons macro would have no target language to expand into. Without the lcons macro, lazy conses can't be used in the above convenient way; we would have to write fib2 in terms of the macro expansion:
$ ./txr -p "(sys:expand '(defun fib2 (a b)
(lcons a (fib2 b (+ a b)))))"
(defun fib2 (a b) (make-lazy-cons (lambda (#:lcons-0001)
(rplaca #:lcons-0001 a) (rplacd #:lcons-0001 (fib2 b (+ a b))))))
The circular list mlet, when fully expanded looks like this, by the way:
It's a gritty oat-meal of delays, lambdas, forces, and cons manipulation. One thing that is conspicuously absent amid the toenail clippings: what happened to the x variable? Haha!
A lisp macro is a code transformer run by the compiler using the full features of the language to generate code.
Macros allow for the arbitrary evaulation of it's arguments (rather than the standard left to right order before a function call), allowing you to do syntactic extensions without the added boilerplate that functional languages can require.
Essentially, each macro allows you to define a mini-language that is parsed by the compiler that returns code that is then compiled. It takes a while to groc, but once you do you can never really go back.
This is only ~90% likely to be correct, since it's second-hand information and I don't use LISP actively.
A LISP macro is a syntax transformation. It lets you write code in the way you want to, instead of whatever level of abstraction you used to have.
I'm not sure about 'partially formed' code output, but 'partially formed' input is definitely possible. The way to invoke a LISP macro need not be valid LISP. In short, LISP macros excel at creating domain-specific languages.
In my Lisp-esque language I use a temporary macro to automate the creation of some similar standard library procedures. This happens at runtime in the stdlib source file that is loaded:
# Define procedures named int? float? etc that test the type of a value.
(def def-type-predicate (mac (type-name)
`(def ,(string-to-symbol (join "" $type-name "?"))
(proc (x) (eq? (type x) ',type-name)))))
(def-type-predicate int)
(def-type-predicate float)
(def-type-predicate bool)
(def-type-predicate string)
(def-type-predicate symbol)
(def-type-predicate file)
(def-type-predicate nil)
(def-type-predicate pair)
(def-type-predicate procedure)
(def-type-predicate macro)
(zap def-type-predicate)
The crucial thing about what happens there being that the (def foo? ...) value produced by each macro invocation then gets evaluated in the root/top-level environment and so results in a "global" procedure definition. Using them:
But it's nice to have single-parameter ones for FP list stuff. I suppose that 2-parameter version could have their order swapped and do partial application on top of it.
Anyway, was just sharing something I had fun making. No language wars intended.
The first order function is evaluated at runtime, every time the program is executed. Macros are expanded at compile time, so they will be calculated just once. Thus the syntactic sugar added by the macro doesn't incur a time penalty.
This can provide an important speed improvement for complex macros or code used often, in tight loops or frequently called functions (it's like using inline methods in .h files in C++).
You can support new paradigms, just as a library. For example, core.async was written as a macro. So Clojure basically got Go channels in a library, not a language upgrade.
Macros can fundamentally change code. Codewalkers. Back when OOP was a big thing (probably still is), Lisp programmers would probably amuse themselves by making object-oriented extensions to the language via macros, and sending them to each other.
This means that you are empowered, not just a language implementer. (First-class functions are important, and typically a better idea than "Guess I'll write my own macro!" but they only go so far.)
The difference is subtle. With high order function you coule define (abstract out) a new idiom, while with proper macros you could define a new special form.
In case of using high-order procedure all its arguments will be evaluated in order before application of the procedure while in case if a macro you could explicitly define all the transformations (evaluation rules) for each argument. This is why it is called a special form - it has its own evalustion rules.
Shortcircuiting if or and are canonical examples.
So, with macros you are extending Lisp with new special forms.
Your question doesn't make sense. High order functions and macros are orthogonal concepts.
Lisp supports first-class, high order functions. In fact, it was probably the first language to do so.
Macros are different. And they're not just "expanded." Think of macros as full blown Lisp functions that run at compile time and generate new Lisp code, that is itself compiled at compile time. Since Lisp code is represented as Lisp's list data structure, it's super easy. The macro system does allow expansion/replacement (like C's preprocessor), but that's just scratching the surface.
Can, say, a Lisp macro be 'partially formed', in the sense that it can expand into some boilerplate that represents an incomplete syntax tree? (Whereas a higher-order-function is necessarily complete.) I can see that being useful, but not inasmuch a it's made out.