How do you compile macros in a Lisp compiler?

Question

In a Lisp interpreter, there can easily be a branch in eval that can expand a macro, and in the process of expanding it, call functions to build up the expanded exp

Answer 1

There are many approaches to this. One extreme is something called "FEXPER", which are macro-like things that essentially get re-expanded on every evaluation. They caused a lot of noise at some point in the past but have almost completely disappeared. (There are a few people who still do similar things though, newlisp is probably the most popular example.)

So FEXPERs were dumped in favor of macros, which are more "well behaved" in a way. You basically do macro expansion once, and compile the resulting code. As usual, there are a few strategies here, which can lead to different results. For example, "expand once" doesn't specify when it gets expanded. This can happen as soon as the code is read, or (usually) when it is compiled, or even just on the first time it runs.

Another question here -- and that's essentially where you stand -- is in what environment you evaluate the macro code. In most Lisps, everything happens in the same happy global environment. A macro can access functions freely, which can lead to some subtle problems. One outcome of this is that many commercial Common Lisp implementations give you a development environment where you do most of your work and compile things -- this makes the same environment available on both levels. (Actually, since macros can use macros, there are an arbitrary number of levels here.) To deploy an application you get a restricted environment that doesn't have, for example, the compiler (ie, the compile function), since if you deploy code that uses that, your code is essentially a CL compiler. So the idea is that you compile the code on your full implementation, and that expands all macros, which means that the compiled code has no additional uses of macros.

But of course that can lead to those subtle problems that I talked about. For example, some side-effects can lead to a loading-order mess, where you need to load code in a specific order. Worse, you could fall into a trap where code runs one way for you, and another way when it's compiled -- since compiled code already had all macros (and the calls they made) expanded beforehand. There are some hackish solutions to these, like eval-when that specifies certain conditions for evaluating some code. There are also a few package systems for CL where you specify things like loading order (like asdf). Still, there is no real robust solution there, and you can still fall into these traps (see for example this extended rant).

There are alternatives, of course. Most notably, Racket uses its module system. A module can be "instantiated" multiple times, and state is unique to each instance. Now, when some module is used in both macros and in runtime, the two instances of this modules are distinct, which means that compilation is always reliable, and there are none of the above headaches. In the Scheme world, this is known as "separate phases", where each phase (runtime, compile-time, and higher levels with macros-using-macros) has separate module instances. For a good introduction to this and a thorough explanation, read Matthew Flatt's Composable and Compilable Macros. You could also just look at the Racket docs, for example, the Compile and Run-Time Phases section.