This post was updated on 9 August 2023 with the sections 'What are Macros in Elixir?' and 'Why You Need Macros'.
Welcome back to part two of this series on metaprogramming in Elixir. In part one, we introduced metaprogramming and gave a brief overview of macros.
In this part, we will explore the inner workings and behaviors of macros in more depth.
Although we had discussed what macros are in the previous post, it doesn't hurt to refresh our memory on what macros are and why they are important.
What are Macros in Elixir?
Macros are powerful compile-time constructs that generate and modify code before it is compiled into bytecode. Elixir macros are used to define new language constructs, extend already existing ones, or perform other code transformations that are necessary for developer productivity.
Why You Need Macros
In a nutshell, Elixir macros are important because:
- They can help you handle abstraction. By writing macros to handle all sorts of specialized functionalities in your app, you'll likely end up with a more maintable code base as your app grows, since complexity is handled properly.
- You can use them to run compile-time code checks, which will help your overall code quality.
- They are perfect for creating Elixir libraries. If you've used a library like Cowboy or even Ecto, then what you see is an example of an Elixir macro. So if your goal is to create an Elixir library to share with the Elixir community or for your own purposes, you will find that macros are the perfect way to go about it.
And with that, we can now see what happens during the compilation process of an Elixir app.
Stages of Elixir's Compilation Process
We can boil down the Elixir compilation process to the following basic stages:
(Note: the actual compilation process of an Elixir program is more intricate than above.)
The compilation process can be broken down into the following phases:
- Parsing — The Elixir source code (program) is parsed into an AST, which we will call the initial AST.
- Expansion — The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a final AST is generated.
- Bytecode generation phase — After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode.
As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work.
Expansion Phase
Let's start by first examining the expansion phase on a general level.
The compiler will expand macros (as per Macro.expand
) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro
until it reaches its most fundamental AST form.
As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time.
If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively.
We can observe this behavior as follows:
defmodule Foo do defmacro foo do quote do IO.inspect("hello") end end end iex(1)> require Foo iex(2)> ast = quote do: Foo.foo {{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true], []} iex(3)> ast |> Macro.expand(__ENV__) {{:., [], [ {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]}, :inspect ]}, [], ["hello"]}
ast
represents the initial AST generated by the compiler before expansion. It holds a reference to the macro
Foo.foo
but it is not expanded as the macro has not been evaluated yet.
When we call Macro.expand
on the given AST, the compiler begins by injecting the behavior of the macro
into the callsite. We can expand the AST one step at a time using
Macro.expand_once
.
Contexts in Macros
Now that we understand the basics of the expansion phase, we can investigate the parts of a macro.
Macros contain two contexts — a macro context and a caller context:
defmacro foo do # This is the macro's context, this is executed when the macro is called # This is the return value of the macro (AST) quote do # This is the caller's context, this is executed when the callsite is called end end
As you can see, the macro's context is any expression declared before the quote
.
The caller's context is the behavior
declared in the quote
. The quote
generated AST is the macro's output and is
injected into and expanded at the callsite.
The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from Metaprogramming Elixir, illustrates this:
defmodule Mod do defmacro definfo do IO.puts "definfo :: Macro's context #{__MODULE__}" quote do IO.puts "definfo :: Caller's context #{__MODULE__}" def friendly_info do IO.puts "friend_info :: Module name #{__MODULE__}" end end end end defmodule MyModule do require Mod Mod.definfo end iex(1)> c "context.exs" definfo :: Macro's context Elixir.Mod definfo :: Caller's context Elixir.MyModule [Mod, MyModule] iex(2)> MyModule.friendly_info friend_info :: Module name Elixir.MyModule :ok
As you can see, the module that executes the macro's context is Mod
. But the module that executes the caller's
context is MyModule
— the callsite where the macro is injected and expanded.
Similarly,
when we declare friendly_info
, we inject this function into the callsite of the macro, which is MyModule
. So the function now 'belongs' to MyModule
.
But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another?
Order of Evaluation in Macro and Caller Contexts
The key difference between the macro context and the caller context is that behavior is evaluated at different times.
Let's look at an example:
defmodule Foo do defmacro foo do IO.puts("macro") quote do IO.puts("caller") end end end iex(1)> require Foo iex(2)> ast = quote do: Foo.foo {{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true], []} iex(3)> ast |> Macro.expand(__ENV__) macro {{:., [], [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]}, [], ["caller"]}
When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite.
However, we can go a level deeper.
Let's look again at the first component of the expansion phase: Macros are executed. Their output (AST) is injected and expanded into the callsite.
For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase.
If we think of macros as regular functions, the macro context is the function body and the caller context is
the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the quote
is
evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the
macro.
The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module.
The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated.
The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate?
Well, we can use another example here:
defmodule Foo do defmacro foo do IO.puts("macro") quote do IO.puts("caller") end end end defmodule Bar do require Foo def execute do IO.puts("execute") Foo.foo end end macro iex(1)> Bar.execute execute caller
The caller context evaluates when the function is called (as we established earlier).
However, something interesting happens with our macro context — it evaluates when the module compiles. This evaluation only happens once when Bar
compiles, as evidenced by the lack of "macro" in our output
when we call Bar.execute
. Why is this the case?
Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called.
This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts.
We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code.
We use the caller context when we want to invoke behavior injected into the callsite at evaluation.
Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit
unquote
.
Revisiting unquote
In part one of this series, we established that unquote
evaluates a given expression and injects the result (as an
AST) into the AST built from quote
. This is only a piece of the puzzle.
Let's dig deeper to understand the behavior of unquote
during compilation and the necessity of using it.
While the rest of the quote
body is evaluated at the same time as the callsite, unquote
is evaluated
(immediately) during compile-time — when the macro is evaluated. unquote
aims to evaluate
and inject the result of a given expression. This expression might contain information that is only available during the
macro evaluation, including variables that are initialized during this process. unquote
must be
evaluated during compile-time along with the macro, so that the AST of the result injects into the quote
that
we build.
But why do we need to unquote
the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using unquote
against one that does not:
defmodule Foo do defmacro foo(x) do quote do IO.inspect(x) end end end defmodule Bar do defmacro bar(y) do quote do IO.inspect(unquote(y)) end end end iex(1)> require Foo iex(2)> require Bar iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3) iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3) iex(5)> ast_foo |> Macro.expand(__ENV__) {{:., [], [ {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]}, :inspect ]}, [], [{:x, [counter: -576460752303423448], Foo}]} iex(6)> ast_bar |> Macro.expand(__ENV__) {{:., [], [ {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]}, :inspect ]}, [], [ {:+, [context: Elixir, import: Kernel], [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]} ]}
Observe that the expanded Foo.foo
AST is vastly different from the Bar.bar
AST even though they are
both given the same variable. This is because Elixir is quite literal with variable references. If a variable is
referenced without unquote
, an AST of that variable reference injects into the AST.
Using unquote
ensures that the underlying AST of the variable's value injects into the quote
body.
Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter?
The scoping of variables in macros can be a confusing subject, so let's demystify it.
Variable Scoping in Macros
Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in
macros, and when to use the options
unquote
and bind_quoted
in quote
.
Due to function clause scoping, the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function. Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros.
When the macro context is evaluated, its arguments and any initialized variables are 'in scope.'
This is why unquote
can evaluate variable references declared as arguments of the macro or any variables
initialized in the macro context.
Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution.
To understand this difference better, let's look at a few examples:
defmacro foo do quote do x = 1 + 1 def bar, do: IO.inspect(unquote(x)) end end
In this first example, unquote
will not work. The variable x
has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of unquote
runs too early, so we cannot reference our
variable x
when we need to. When unquote
evaluates during compile-time, it attempts to evaluate the variable
reference expression of x
and finds that it is not in scope.
How can we fix this? By disabling unquoting. This
means disabling the immediate evaluation of unquote
. We only want unquote
to evaluate when our caller context evaluates. This ensures that
unquote
can properly reference a variable in scope (x
) as variable initialization would have occurred
during the evaluation of the callsite.
defmacro foo do quote unquote: false do x = 1 + 1 def bar, do: IO.inspect(unquote(x)) end end
This example highlights the impact of scoping in macros. If we attempt to access a variable that is available
during the evaluation of the macro context, unquote
as-is is perfect for us.
However, suppose we try to access a
variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior
to initialize variables in scope before unquote
attempts to reference them.
Let's apply this understanding to two other examples.
defmacro foo(opts) do quote bind_quoted: [opts: opts] do x = Keyword.get(opts, :x) def bar, do: IO.inspect(unquote(x)) end end
In this example, we have initialized the variable x
from a keyword list. As the keyword list is initialized
during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to:
- Generate an initialization of the variable during the evaluation of the callsite, and
- Disable unquoting behavior.
We have to bind opts
to the caller context, as the variable is no longer in scope during the evaluation of the
callsite.
Finally, we have:
defmacro foo(x) do quote do def bar, do: IO.inspect(unquote(x)) end end
In this last example, x
remains a variable in scope during the evaluation of the macro context — i.e. when
the macro is called. The immediate evaluation of unquote
works in our favor. It
renders unquote(x)
valid, as x
is in scope when unquote
is evaluated.
Macro Hygiene in Elixir
While we are on the topic of scopes in macros, let's discuss macro hygiene.
According to tutorialspoint.dev:
Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.
This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables.
Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the official tutorial.
A macro variable is declared within the body of quote
, while a caller variable is declared
within the callsite of the macro.
defmodule Foo do defmacro change do quote do: a = 13 end end defmodule Bar do require Foo def go do a = 1 Foo.change a end end Bar.go # => 1
In this example, a
referenced in change
is not the same variable a
in the scope of go
. Even when we
attempt to change the value of a
, the value of a
in the scope of go
remains untouched.
However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope.
Elixir provides the macro var!
to bridge the gap between these two scopes:
defmodule Foo do defmacro change do quote do: var!(a) = 13 end end defmodule Bar do require Foo def go do a = 1 Foo.change a end end Bar.go # => 13
This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to).
You can apply the same hygiene to aliases and imports.
Understanding require
In the code examples shown in this section, we have always used require <module>
before we invoke the macros
within the module. Why is that?
This is the perfect segue into how the compiler resolves modules containing macros — particularly the order in which modules are compiled.
In Elixir, modules compile in parallel, and usually — for regular modules — the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations.
However, as macros must be available during compile-time, the module these macros
belong to must be compiled beforehand.
Here's where require
comes into the picture. require
explicitly informs the compiler to compile and load the module
containing the macro first.
We can use an example to illustrate this behavior:
defmodule Foo do IO.puts("Compiling Foo") defmacro foo do IO.puts("Foo.foo := macro") quote do IO.puts("Foo.foo := caller") end end end defmodule Bar do IO.puts("Compiling Bar") require Foo IO.puts("Bar := before Foo.foo") Foo.foo() end Compiling Foo Foo.foo := macro Compiling Bar Bar := before Foo.foo Foo.foo := caller
(Note that this is just an approximation of the
actual compilation process, but it aims to paint a clearer picture of how require
works.)
Let's try to understand why the outputs are in this order.
Firstly, Bar
tries to compile. The compiler scans and finds a require
for Foo
before evaluating any module-level expressions within the module (such as IO.puts
). So it pauses the compilation of
Bar
and compiles the Foo
module first. As Foo
is compiled, module-level code — like IO.puts
— is
evaluated, and the compiler prints the first line of the output.
Once Foo
is compiled, the compiler returns to Bar
to resume compilation. Bar
is parsed, macro calls are
executed, and the macro context is evaluated. Even though Foo.foo
is called after
IO.puts("Bar := before Foo.foo")
, the evaluation of the macro call takes precedence over the evaluation of
module-level code.
During expansion, Foo.foo
's caller context is injected and expanded into the callsite in Bar
. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration.
In essence, require
instructs the compiler on the order of compilation that each module should go through if there are
macro dependencies. This ensures that the macros are available during compile-time.
Escaping Macros in Elixir
Before explaining what Macro.escape
does, let's look at an example:
iex(1)> x = {1, 2, 3} iex(2)> quote do: unquote(x) {1, 2, 3} iex(3)> ast = quote do: IO.inspect(unquote(x)) {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]} iex(4)> Code.eval_quoted(ast) ** (CompileError) nofile: invalid quoted expression: {1, 2, 3} Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3 (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3
That is a strange error. Based on our understanding of unquote
and macros, the code should work as intended,
but it doesn't. Why is that?
Well, the answer is found on iex(2)
. When we attempt to unquote x
, we return not an AST, but a tuple — the
same one initially assigned to x
. The error then points to the fact that the tuple is an invalid quoted
expression.
When we
unquote(x)
as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a
valid AST and throws an error.
So, how do we fix it?
We need to convert the raw tuple referenced by x
into a valid AST. This can be achieved by
escaping this value using Macro.escape
. Let's understand what Macro.escape
does:
iex(1)> a = {1, 2, 3} {1, 2, 3} iex(2)> Macro.escape(a) {:{}, [], [1, 2, 3]} iex(3)> quote do: unquote(a) {1, 2, 3} iex(4)> quote do: unquote(Macro.escape(a)) {:{}, [], [1, 2, 3]}
In iex(2)
, we see that Macro.escape(a)
returns an AST of the tuple, not the raw tuple — and this is exactly what we
are looking for. By combining Macro.escape
's behavior with unquote
, we can inject the AST of the tuple into
the quote
as seen in iex(4)
.
Let's test this:
iex(1)> x = {1, 2, 3} {1, 2, 3} iex(2)> quote do: unquote(Macro.escape(x)) {:{}, [], [1, 2, 3]} iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x))) {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{:{}, [], [1, 2, 3]}]} iex(4)> Code.eval_quoted(ast) {1, 2, 3} {{1, 2, 3}, []}
As you can see, the code works just as intended because we escape the tuple.
Often when working with data structures like tuples and dictionaries, you may find that the injected data from unquote
does not inject a valid AST. In these cases, you should use Macro.escape
before unquote
.
Guard Clauses and Pattern Matching
Finally, it's worth mentioning that, much like regular functions defined through def
, macros can use guard
clauses and pattern
matching:
defmacro foo(x) when x == 4 do IO.inspect("macro with x being 4") end defmacro foo(_) do IO.inspect("macro with any other value") end
Up Next: Applications of Macros in Elixir
Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally.
In part three, we will look at the many applications of macros in Elixir.
Happy coding!
P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!