How does macroexpansion actually work in Lisp?

Question

I would like a more detailed explanation of how macro expansion works, at least in Emacs Lisp but an overview of other Lisps would be appreciated. The way I usually see it explained is that the arguments of the macro are passed unevaluated to the body, which is then executed and returns a new LISP form. However, if I do

(defun check-one (arg)
    (eq arg 1))

(defmacro check-foo (checker foo)
    (if (checker 1)
        `(,foo "yes")
      `(,foo "no")))

I would expect

(check-foo check-one print)

to first expand to

(if (check-one 1)
        `(print "yes")
        `(print "no))

and then finally to (print "yes") but instead I get a "checker" function is void error. On the other hand, if I had defined

(defmacro check-foo (checker foo)
    (if (funcall checker 1)
        `(,foo "yes")
      `(,foo "no")))

then I would have the expected behavior. So the expressions do get replaced in the body unevaluated, but for some reason functions do not work? What is the step-by-step procedure the interpreter follows when macroexpanding? Is there a good text-book that explains this rigorously?

Answer 1

Macros are functions ...

A good way to think about macros is that they are simply functions, like any other function.

... which operate on source code

But they are functions whose arguments are source code, and whose value is also source code .

Looking at macro functions

Macros being functions is not quite explicit in elisp: some of the lower-level functionality is, I think, not exposed. But in Common Lisp this is quite literally how macros are implemented: a macro has an associated function, and this function gets called to expand the macro, with its value being the new source code. For instance, if you are so minded you could write macros in Common Lisp like this.

(defun expand-fn (form environment)
  ;; not talking about environment
  (declare (ignore environment))
  (let ((name (second form))
        (arglist (third form))
        (body (cdddr form)))
    `(function (lambda ,arglist
                 (block ,name
                   ,@body)))))

(setf (macro-function 'fn) #'expand-fn)

And now fn is a macro which will construct a function which 'knows its name', so you could write

(fn foo (x) ... (return-from foo x) ...)

which turns into

(function (lambda (x) (block foo ... (return-from foo x))))

In Common Lisp, defmacro is then itself a macro which arranges for a suitable macro function to be installed and also deals with making the macro available at compile time &c.

In elisp, it looks as if this lower layer is not specified by the language, but I think it's safe to assume that things work the same way.

So then the job of a macro is to take a bunch of source code and compute from it another bunch of source code which is the expansion of the macro. And of course the really neat trick is that, because source code (both arguments and values) is represented as s-expressions, Lisp is a superb language for manipulating s-expressions, you can write macros in Lisp itself.

Macroexpansion

There are a fair number of fiddly corner cases here such as local macros and so on. But here is, pretty much, how this works.

Start with some form <f> :

1. If <f> is (<a> ...) where <a> is a symbol, check for a macro function for <a> . If it has one, call it on the whole form, and call the value it returns <f'> : now simply recurse on <f'> .
2. If <f> is (<a> ...) where <a> is a symbol which names a special operator (something like if ) then recurse on the subforms of the special operator which its rules say must be macroexpanded. As an example, in a form like (if <x> <y> <z>) all of <x> , <y> , & <z> need to be macroexpanded, while in (setq <a> <b>) , only <b> would be subject to macroexpansion, and so on: these rules are hard-wired, which is why special operators are special.
3. If <f> is (<a> ...) where <a> is a symbol which is neither of the above cases, then it's a function call, and the forms in the body of the form are macroexpanded, and that's it.
4. If <f> is ((lambda (...) ...) ...) then the forms in the body of the lambda (but not its arguments!) are macroexpanded and then the case is the same as the last one.
5. Finally <f> might not be a compound form: nothing to do here.

I think that's all the cases. This is not a complete description of the process because there are complications like local macros and so on. But it's enough I think.

Order of macroexpansion

Note that macroexpansion happens 'outside in': a form like (a ...) is expanded until you get something which isn't a macro form, and only then is the body, perhaps, expanded. That's because, until the macro is completely expanded, you have no idea which, if any, of the subforms are even eligible for macroexpansion.

---

Your code

My guess is that what you want to happen is that (check-foo bog foo) should turn into (if (bog 1) (foo yes) (foo no)) . So the way to get this is that this form is what the macro function needs to return. We could write this using the CL low-level facilities:

(defun check-foo-expander (form environment)
  ;; form is like (check-foo pred-name function-name)
  (declare (ignore environment))        ;still not talking about environment
  `(if (,(second form) 1)
       (,(third form) "yes")
     (,(third form) "no")))

And we can check:

> (check-foo-expander '(check-foo bog foo) nil)
(if (bog 1) (foo "yes") (foo "no"))

And then install it as a macro:

> (setf (macro-function 'check-foo) #'check-foo-expander)

And now

> (check-foo evenp print)

"no" 
"no"

> (check-foo oddp print)

"yes" 
"yes"

But it's easier to write it using defmacro :

(defmacro check-foo (predicate function)
  `(if (,predicate 1)
       (,function "yes")
     (,function "no")))

This is the same thing (more-or-less), but easier to read.