In the first part of this series on maintainable Elixir code, we started by applying rules for code predictability in our code design. We immediately saw an emerging pattern: a series of transformations on state. In this part, we'll explore this pattern further.
We'll first learn how to write expressions as reducers. Then we'll use metaprogramming to make use of reducers and enforce code style seamlessly.
Finishing up, we'll see an example where all the pieces fit together.
Let's get going!
Quick Recap of Part One and What We'll Do
In the first part, we experimented with applying code styles to this snippet:
with %Payment{} = payment <- fetch_payment_information(params), {:ok, user} <- Session.get(conn, :user), address when !is_nil(address) <- fetch_address(user, params), {:ok, order} <- create_order(user, payment, address) do conn |> put_flash(:info, "Order completed!") |> render("checkout.html") else {:error, :payment_failed} -> handle_error(conn, "Payment Error") %Store.OrderError{message: message} -> handle_error(conn, "Order Error") error -> handle_error(conn, "Unprocessable order") end
By the end of this post, we'll have all the tools to rewrite it like this:
case Checkout.execute(%{params: params}, options) do {:ok, %{order: order}} -> conn |> put_flash(:info, "Order completed!") |> redirect(to: Routes.order_path(conn, order)) {:error, error} conn |> put_flash(:error, parse_error(error_description)) |> render("checkout.html") end
While writing this post, I created ex_pipeline, an open source library that allows developers to create pipelines while enforcing code style easily.
All code examples in this post are simplified versions of what exists in that project.
Chain of Transformations
One of my favorite hobbies is woodworking — I'm not good at it, by all means, but it's something that I do enjoy. Before starting a new project, I always write down all the materials and tools I'll need, as well as the steps I'll follow. That's how I go from pieces of wood to a finished table, from an initial state to a finished state.
Let's think about how a table is made: you get some raw materials — wood, nails, glue — and a few tools. Then you execute a set of actions to modify the shape and appearance of the materials — cutting, assembling, polishing, and painting.
You apply a chain of transformations on an initial state (raw materials) until you get the desired state (the table).
We already talked about the emerging pattern of a pipeline. That's what we are going to explore over the next paragraphs.
Writing Expressions as Reducers in Elixir
Reducers are one of the key concepts of functional programming. They are very simple to understand: you take a list of values, apply a function to each element, and accumulate the results.
In Elixir, we can work with reducers using the Enum.reduce/3
function and its variants — Enum.reduce/2 and
Enum.reduce_while/3.
For example, we can use a reducer to filter all even numbers from a list:
numbers = [1, 2, 3, 4, 5, 6] starting_value = [] Enum.reduce(numbers, starting_value, fn current_number, filtered_values -> if rem(current_number, 2) == 0 do # is the current number even? [number | filtered_values] # if so, add it to the accumulator else # otherwise... filtered_values # just ignore it and return the accumulator end end)
This code will return the list [2, 4, 6].
To execute different actions on a value, we can transform the list of values into a list of functions. Then we pass the value as the initial value for the accumulator:
transformations = [ fn x -> x + 1 end, # add 1 fn x -> x * 2 end # multiply by two ] starting_value = 10 Enum.reduce(transformations, starting_value, fn transformation, current_value -> apply(transformation, [current_value]) end)
This looks like a complicated way to write (10 + 1) * 2, right? But by expressing each transformation as a
function, we can attach an arbitrary number of transformations. They all must accept the same number of parameters and return the next updated state. In other words, these functions must look the same and be small.
In the end, we want to write the previous example as something like this:
defmodule MyModule do use Pipeline def sum_step(value, _), do: {:ok, value + 1} def double_step(value, _), do: {:ok, value * 2} end
In the next part, we can do just that using a bit of magic from macros!
Sewing Reducers Into Features with Macros
Now that we know how to use reducers let's use them to write pipelines.
The goal is to make the process of writing, reading, and maintaining pipelines as easy as possible. So, instead of adding a DSL or using complex configs, we just want to write our functions using the same pattern of name and arity. We can then automate the process of building the pipeline by detecting which functions follow this pattern.
We create a new pipeline by writing code on the same pattern.
The State: Track What's Happening in Elixir
Before we dive into macros, let's take a step back and talk a little bit about the state of the pipeline.
When executing a pipeline, we need to know a few things: the initial state, the current state, if it's still valid, and if there are any errors. All this information will help us debug and troubleshoot any problems we might find when writing our code.
Also, by abstracting state updates in this module, we can enforce a very important rule on reducers: the return value must be an ok/error tuple.
The starting point of a module that manages state should be something like this:
defmodule Pipeline.State do defstruct [:initial_value, :value, :valid?, :errors] def new(value) do %Pipeline.State{valid?: true, initial_value: value, value: value, errors: []} end def update(state, {module, function_name} = _function_definition, options \\ []) do case apply(module, function_name, [state.value, options]) do {:ok, value} -> %Pipeline.State{state | value: value} {:error, error_description} -> %Pipeline.State{state | valid?: false, errors: [error_description | state.errors]} bad_return -> raise "Unexpected return value for pipeline step: #{inspect(bad_return)}" end end end
The new/1 function will create a new valid and clean %Pipeline.State{} struct based on the initial value we
give to the pipeline executor.
The update/3 function will update or invalidate the given state by calling function with the state's value and
the given options.
If the given function returns an {:ok, <updated_value>}, then the state's value is updated,
and we are good to go. However, if the function returns an {:error, <error>} value, the state is invalidated. The returned error is added to the list of errors in the state. If the function returns anything that's different from an
ok/error tuple, it will raise an exception.
See the full implementation of this module.
Pipeline Steps and Hooks
Each function that transforms the state will be called a step. Here are some guidelines to follow:
- A step is a function whose name ends with
_stepand accepts two parameters. - Steps execute in the order they are declared in their module.
- If one step fails, then all remaining steps are ignored.
Sometimes, we still want to execute code, even during failures — write a log, update a trace, publish a message, etc. We have a special type of step for these cases: a hook. Let's list the guidelines for hooks:
- A hook is a function whose name ends with
_hookand accepts two parameters. - Hooks execute in the order they are declared, just like steps.
- They are always executed at the end of the pipeline, after all steps execute.
Good! But how do we detect that steps and hooks exist? The answer is metaprogramming! Luckily for us, Elixir has powerful metaprogramming tools to make all this possible.
Setting Up the Required Macros in Elixir
We'll need two things: macros and compile callbacks to read information about the functions of a module, and store what a step and a hook are.
Starting with the base module Pipeline, we want to use Pipeline in other modules. For that to happen, we
declare a special macro, __using__/1. This macro is called with the keyword use in Elixir.
In this macro, we add a compile hook to the target module so that we can inject code into it. The @before_compile
hook does exactly that by calling the given function when a compiler is about to generate the underlying bytecode. It
accepts either a module or a tuple with a module and function name. We'll go with the second option to make things simpler.
defmodule Pipeline do defmacro __using__(_options) do quote do # Calls Pipeline.build_pipeline/1 before generating the final bytecode, allowing us to analyze, generate and # inject code whenever we call `use Pipeline`. @before_compile {Pipeline, :build_pipeline} end end defmacro build_pipeline(env) do # inject code! end end
Anything called inside the quote block will be injected into the caller module — including the result of the
@before_compile hook.
Check out the official docs for more information about quote and unquote blocks.
Now, whenever we want to create a pipeline, we call use:
defmodule MyPipeline do use Pipeline # <- code will be injected! end
Detecting Steps and Hooks
We have everything in place to inject code, but we still need to generate the code that will be injected. Let's expand
the build_pipeline/1 macro:
defmacro build_pipeline(env) do # Fetch all public functions from the caller module definitions = Module.definitions_in(env.module, :def) # Filter functions that are steps and hooks steps = filter_functions(env.module, definitions, "_step", 2) hooks = filter_functions(env.module, definitions, "_hook", 2) # Generate code on the caller module quote do # returns all steps and hooks def __pipeline__, do: {unquote(steps), unquote(hooks)} # Syntatic sugar so we can execute directly from the module that defines it def execute(value, options \\ []), do: Pipeline.execute(__MODULE__, value, options) end # Filter functions from the given module based on their suffix and arity defp filter_functions(module, definitions, suffix, expected_arity) do functions = Enum.reduce(definitions, [], fn {function, arity}, acc -> # Detect if the function name ends with the desired suffix: _step or _hook valid_name? = function |> Atom.to_string() |> String.ends_with?(suffix) # Detect if the function arity matches the expected arity has_expected_args? = arity == expected_arity cond do valid_name? and has_expected_args? -> # In this case, the function does end with the desired suffix and has the correct arity. # We include it in the accumulator along with the line where it was declared so we can # order them by their position in their module {_, _, [line: line], _} = Module.get_definition(module, {function, arity}) [{module, function, line} | acc] valid_name? -> # If the function has the desired suffix but the arity is not correct, we raise this error # because we don't want people trying to figure out why their steps or hooks are not # being executed. raise(PipelineError, "Function #{function} does not accept #{expected_arity} parameters.") true -> # If the function doesn't match our filters, it's not a part of the pipeline and can be # just ignored acc end end) # At this point we have filtered the functions that match the filter criteria but they # are not in order. We use the line number information to sort them and then drop it # once we don't need it anymore. functions |> Enum.sort(fn {_, _, a}, {_, _, b} -> a <= b end) # order by line number |> Enum.map(fn {m, f, _l} -> {m, f} end) # drop line number end end
This macro receives the compiler env, which has lots of context information. Here, the important bit is the caller module,
accessible via the env.module attribute.
Then we use the module.definitions_in/2 special
function to get a list of all functions declared on the module env.module with the def keyword.
We call the
Pipeline.filter_functions/4 function to filter these definitions by their suffix and arity, essentially detecting our
steps and hooks! The Pipeline.filter_functions/4 function is kind of big, so I've added comments to help you
navigate through it.
As said before, anything inside the quote block is injected directly into the caller module. So whenever we call
use Pipeline, the module will have two extra functions: __pipeline__/0, and execute/2.
The Pipeline module uses the first function to execute our custom pipeline, while the execute/2
function is just a convenience function that will execute Pipeline.execute/3. It allows us to execute our pipeline by
calling MyPipeline.execute(value, options).
Speaking of Pipeline.execute/3, it's time to define it. This function is the core of this engine, and it's responsible
for actually calling a reducer that will power our pipeline engine:
defmodule Pipeline do # ... def execute(module, value, options) do # Build a new initial state of the pipeline initial_state = State.new(value) # Fetch steps and hooks from the module {steps, hooks} = apply(module, :__pipeline__, []) # Run each step from the pipeline and build the final state final_state = Enum.reduce(steps, initial_state, fn reducer, current_state -> State.update(current_state, reducer, options) end) # Run each hook with the final state Enum.each(hooks, fn {module, function} -> apply(module, function, [final_state, options]) end) # Transform the state value into an ok/error tuple case final_state do %State{valid?: true, value: value} -> {:ok, value} %State{errors: errors} -> {:error, errors} end end end
Check out the complete version of this module here.
Note: In the complete version of this module, there is also another kind of hook, called an async hook. Async hooks work just like the hooks presented here, but are executed in parallel rather than sequentially. They will not be covered in this post, since they are essentially a hook executed with a Task.
Building and Executing a Pipeline
By simply calling use Pipeline and writing function names with the _step or _hook suffix, we are now able to
build our pipelines. Let's rewrite our last example from the first part of the post with this new mechanic:
defmodule Checkout do use Pipeline def fetch_payment_information_step(value, options) do # ... end def fetch_user_step(value, options) do # ... end def fetch_address_step(value, options) do # ... end def create_order_step(value, options) do # ... end def report_checkout_attempt_hook(state, options) do # ... end end
We can execute this new pipeline like this:
case Checkout.execute(%{params: params}, options) do {:ok, %{order: order}} -> conn |> put_flash(:info, "Order completed!") |> redirect(to: Routes.order_path(conn, order)) {:error, error} conn |> put_flash(:error, parse_error(error_description)) |> render("checkout.html") end
No more with blocks, and no need to manually handle errors. The controller is now focused on handling the HTTP layer, and
the checkout feature has an official place to live. Adding a new step to the checkout process just requires that we
write a function in the right place!
Wrap Up
In this post, we learned how to express a set of transformations as a reducer. Then, using the power of macros, we automatically created pipelines by detecting the signature of functions at compile time.
The features that use pipeline mechanics are explicit and contained — we know exactly what to expect and where to go when modifying them. We expect all steps to look and behave in the same way. The basics of error handling are already in place, you just need to decide what to do. These pipelines are, indeed, predictable.
I hope you've found this two-part series about keeping your Elixir code maintainable helpful. Until next time, happy coding!
P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!
