Elixir Alchemy by AppSignal

Debugging Slow Ecto Queries with AppSignal

Roy Tomeij — 2026-04-02T00:00:00+00:00

A sports car can only be driven as fast as the road it's driven on. If you're stuck behind a tractor on a single-lane road, you're not going anywhere fast. The same idea applies to web performance: your application's throughput is only as fast as it's slowest bottleneck.

For Phoenix applications, that bottleneck is almost always the database. Ecto provides powerful tools that allow us to compose large, readable queries by chaining functions together, but this readability can come at the cost of expensive queries under the hood. Slow queries, unindexed tables, and N+1 requests affect your app's performance.

Finding and resolving these issues can take a lot of work, but as the title suggests, AppSignal can be the roadside assistance you need to get your app back on the road. We'll first look at how AppSignal detects poorly performing queries, then look at understanding why those queries are slow, and then finally fixing the problems.

Finding the Slow Query in AppSignal

It's one thing to know your app has a traffic jam, and quite another to know where that traffic jam exists. Because AppSignal auto-instruments Ecto queries, it's able to both pinpoint where in your app that bottleneck is occurring as well as point out the problematic query itself.

To demonstrate how AppSignal helps identify bottlenecks, we’ll use an example app that mimics a basic social media site, with users, followers and posts. When visiting a user's profile page (/users/:username) the app displays counts of followers and those they follow, who they follow and who follows them, and finally a list of posts from those they follow.

The example app and code used in this article can be found on the GitHub Social app repo. Basic instructions for installing AppSignal in Elixir apps, and a more complete guide are also available.

Although the page appears to be functioning correctly, AppSignal has alerted us to a performance issue that hasn't yet risen to the point of user complaints, and it's done so in three different places under the Performance navigation section:

Issue list
Actions
Slow queries

Issue list, as the name implies, lists issues your app is currently experiencing. In our social app, we can see that our /users/:username endpoint is already listed.

The Actions tab provides a list of routes along with their duration (mean and P90) and throughput. Using this page, you can sort routes based on their duration, and then click into the laggard routes to find more information. Clicking into our /users/:username route reveals that there is an open "Performance incident."

Lastly, we can see which queries are causing the most problems under the Slow queries tab. By default, AppSignal ranks queries by their impact (i.e. frequency * duration) to performance from greatest to least. You'll note that what is listed is the SQL queries themselves, not the Ecto functions or from where in the app they originate.

Understanding Why It's Slow

It's not always easy to track performance issues, but the Samples tool in AppSignal simplifies the process by mapping out each operation in the Timeline view. The visualization it provides enables you to pinpoint the specific Ecto calls responsible for bottlenecks.

You can get to this page by first navigating to the Issue list under the Performance navigation section, and clicking on an "action" you want to explore. From there, click on the specific sample you're interested in from the Samples section.

Once you're on the Samples page, scroll down to the Timeline section. You should see something like this:

What's most noticeable in the timeline is:

Three queries are greater than or equal to 1 ms
One query takes up the majority of the time
There are six queries, but the code (not shown) only calls five functions (indicating a potential N+1 query)

We can get more information on each of the queries by hovering over any of the events. If it's a performance issue, we can then use the explain function common in most relational databases to investigate what's happening.

Running explain analyze on the SQL shown above results in the "Query Plan" shown below:

                                                          QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=106.72..107.47 rows=300 width=127) (actual time=2.782..2.894 rows=1500 loops=1)
   Sort Key: p0.inserted_at DESC
   Sort Method: quicksort  Memory: 275kB
   ->  Hash Join  (cost=2.17..94.37 rows=300 width=127) (actual time=0.496..1.873 rows=1500 loops=1)
         Hash Cond: (f1.followed_id = u2.id)
         ->  Hash Join  (cost=1.07..91.66 rows=300 width=91) (actual time=0.164..1.102 rows=1500 loops=1)
               Hash Cond: (p0.user_id = f1.followed_id)
               ->  Seq Scan on posts p0  (cost=0.00..82.00 rows=2000 width=83) (actual time=0.105..0.504 rows=2000 loops=1)
               ->  Hash  (cost=1.06..1.06 rows=1 width=8) (actual time=0.027..0.028 rows=3 loops=1)
                     Buckets: 1024  Batches: 1  Memory Usage: 9kB
                     ->  Seq Scan on followers f1  (cost=0.00..1.06 rows=1 width=8) (actual time=0.019..0.020 rows=3 loops=1)
                           Filter: (follower_id = 30)
                           Rows Removed by Filter: 3
         ->  Hash  (cost=1.04..1.04 rows=4 width=44) (actual time=0.308..0.309 rows=4 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 9kB
               ->  Seq Scan on users u2  (cost=0.00..1.04 rows=4 width=44) (actual time=0.012..0.013 rows=4 loops=1)
 Planning Time: 0.315 ms
 Execution Time: 3.360 ms

Two things stand out in this plan:

We're scanning many more records than necessary (Seq Scan on posts...rows=2000).
The estimated rows (300) vs the actual rows (1,500) are significantly different which can lead to the database choosing the wrong strategy.

A potential third item might be the Filter.

Let's see what we can do to improve the performance.

Common Ecto Fixes

In his article, Tackling Performance Issues in Ecto Applications, Marcos Ramos addresses several issues affecting performance. The two we're most interested in are "Inefficient Queries in Ecto" and "The N+1 Query Problem".

Inefficient Queries in Ecto

Not all performance wins require the same level of effort. When tackling inefficiencies in Ecto queries, consider the following possible optimizations:

Ensure the tables are indexed correctly
Only ask for what you need
Offload the heavy lifting to views or processes that are better able to handle the work

Validating tables are properly indexed is a great place to start. Adding indexes which should have existed from the beginning (e.g. on foreign keys, sortable fields, and commonly filtered fields) will always bring an improvement to performance. In our example, indexes on posts.user_id,followers.follower_id, followers.followed_id, and posts.inserted_at have all been erroneously forgotten.

The fix is simply creating those indexes with an Ecto migration:

defmodule Social.Repo.Migrations.AddMissingIndexes do
  use Ecto.Migration

  def change do
    create index(:posts, [:user_id])
    create index(:posts, [:inserted_at])
    create index(:followers, [:follower_id])
    create index(:followers, [:followed_id])
    create index(:followers, [:inserted_at])
  end
end

Note: You would normally create a migration for each index or at least group them by affected table.

After making sure your tables are properly indexed, another great tip is to check for “unbound queries”. As the name suggests, unbound queries are those queries which have no constraints. Examples might include queries that don't limit the number of returned rows, aren't constrained by a where/3 clause, or don't use select/3 to limit the fields returned.

As we saw above, our query plan showed that we were working with 2,000 results. We don't need to see that many posts at once, so it makes sense to limit the number of posts per "page". To that end, we'll add both limit/3 and offset/3 to our query below.

def list_feed_posts(user_id, page \\\\ 1, page_size \\\\ 20) do
  from(p in Post,
    join: f in Social.Users.Follower,
    on: f.follower_id == ^user_id and f.followed_id == p.user_id,
    join: u in assoc(p, :user),
    order_by: [desc: p.inserted_at],
    preload: [user: u],
    limit: ^page_size,
    offset: ^page_size * (^page - 1)
  )
  |> Repo.all()
end

If you are still seeing performance issues after indexing tables and constraining the data returned, you may need to consider moving your more complex queries into materialized views or to a background job. Aggregated stats, dashboard metrics, multi-join filtering, long-running reports, and data-exports are all examples of queries or processes which can take advantage of these types of enhancements.

The N+1 Query Problem

As Marcos explains in his article, N+1 queries occur "when an application loads a parent record and its associated child records in separate queries." In our app that might be by first pulling a list of people followed and then retrieving their posts in separate queries. The 1 represents the initial query, while the N represents each consecutive query.

In our example app, what originally looked like an N+1 query was the result of a single function issuing two count queries, and doesn't need improvement:

def get_user_follow_counts(user_id) do
  follower_count =
    from(f in Follower,
      where: f.followed_id == ^user_id,
      select: count(f.id)
    )
    |> Repo.one()

  following_count =
    from(f in Follower,
      where: f.follower_id == ^user_id,
      select: count(f.id)
    )
    |> Repo.one()

  %{followers: follower_count, following: following_count}
end

If you want to understand N+1 queries more, Sapan Diwakar has a great tutorial on how to Find and Fix N+1 Queries Using AppSignal.

Verifying Our Changes Worked

Once you have found and eliminated the speed bumps slowing you down, you'll want to verify that your changes actually fixed the problems. You'll first want to verify your changes locally, but once you've deployed, make sure to review your project's AppSignal page to confirm that your changes are having the effect you're expecting. You can do that by navigating to the route from the Actions page.

Note the Deploy markers below the timeline. These are trivial to set up and will help you identify when changes took effect, and which commit.

You can even drill into the latest Samples to see how your changes affected specific queries against production data.

Wrap-Up

Sports cars are loads of fun to drive on long straightaways, around tight curves, and sometimes even on unpoliced country roads. They're less fun to drive in bad weather or through construction areas. It's a similar experience for your app's users. Speed matters and if your app isn't responsive enough, your users may decide to get off at the next exit.

To avoid that, we first looked at how to use AppSignal to detect and find slow queries, primarily focused on the Performance navigation section and the tools under it: Issue list, Actions, and Slow queries. Next, we used the Timeline view within an individual "sample" to isolate potential causes. This gave us a closer look at the request lifecycle, exposing N+1 queries and the underlying SQL responsible for the lag. Finally, we reviewed common fixes (adding indexes, constraining queries, etc.) and how to verify our fixes worked.

Phoenix gives you speed by default. Don't let a bad query or a poorly indexed table turn your sports car into an old clunker.

Domains and Resources in Ash for Elixir

Roy Tomeij — 2026-01-13T00:00:00+00:00

If you’ve been on the fence about trying the Ash Framework, this post is for you.

We’re going to explore why Ash is a game changer for Elixir developers by building something complex, real, but surprisingly elegant.

We’ll create AshTherapy, an AI-powered therapeutic chat service that handles conversations, messages, and user interactions — all powered by Ash.

Before we go further, let’s first answer the question: what exactly is Ash?

What Is Ash for Elixir?

Ash is a business domain modeling framework for Elixir. It works seamlessly with Phoenix to simplify the part of our app that deals with business logic — our data, relationships, and rules.

Think of it like this:

Phoenix handles the technical layer — HTTP, sockets, and controllers.
Ash handles the business layer — entities, relationships, and actions.

Together, they help us build fast and model correctly, with less boilerplate and more focus on what really matters.

The AshTherapy Use Case

Now that we know what Ash is, let’s see it in action. We’ll bring the concepts of domains and resources to life through our demo app, AshTherapy.

Here’s how it works:

A user visits the site and starts a new conversation.
They set a goal or motive — for example, “I want to handle work stress better.”
They then exchange messages with an AI companion that responds thoughtfully.
This back-and-forth continues, forming a guided digital therapy session.

To model this in Ash, we’ll define three key resources:

User — the person using the service
Conversation — the therapy session container
Message — each exchange in the session (user or AI)

All three live inside a single domain: AshTherapy.Therapy.

In this first post, we’ll focus on defining these resources and understanding how the domain ties them together.

Check Out the Code

Before we dive deeper, a quick note. In this post, we won’t walk through how to create an Ash domain or resource step-by-step — that part is already well covered in the official documentation.

Our focus here is on understanding what domains and resources mean in Ash and how they work together, which is even more important when we’re just getting started.

If we check out the AshTherapy codebase (branch: post-1), we’ll find everything set up and ready to follow along with the examples in this post. We’ll take time here to explain the code in that project.

To learn how to create a new domain or resource from scratch, we can refer to the official Ash HexDocs.

Create a new domain using:

mix ash.generate.domain MyApp.MyDomain

Create a new resource using:

mix ash.generate.resource MyApp.MyDomain.MyResource

With that out of the way, let’s explore the User, Conversation, and Message resources that make up our AshTherapy domain.

Ash Domains

An Ash domain is a logical container that groups related resources together. It keeps our business logic organized and provides a clean boundary for everything that belongs to a specific part of our application — like Therapy, Accounts, or Billing.

While domains might seem simple at first, they become much more powerful as our app grows. That’s because many of Ash’s advanced features — like policies, code interfaces, and automated APIs — are built on top of domains.

We’ll create a single AshTherapy.Therapy domain, which looks like this:

defmodule AshTherapy.Therapy do
  use Ash.Domain

  resources do
    resource AshTherapy.Therapy.User
    resource AshTherapy.Therapy.Conversation
    resource AshTherapy.Therapy.Message
  end
end

We use the resources DSL to wrap the individual resources managed by this Ash domain.

Understanding Ash Resources

An Ash resource represents a business entity — such as a User, Conversation, or Message in our AshTherapy app. Together with domains, resources form the business engine that powers our application.

Every resource in Ash is just a normal Elixir module — until we add this line:

use Ash.Resource

That one line turns a plain module into a powerful declarative resource. It gives us access to Domain Specific Language (DSL) blocks like:

attributes: Define fields and data types
relationships: Define associations between resources
actions: Define what operations can be performed

There are many other DSLs available, such as policies and identities, but in this post, we’ll focus on these three foundational blocks.

Resource Structure

Like Ash domains, resources are also plain Elixir modules that consume macros from the Ash library, which grant them their superpowers. The basic structure of a resource looks like this:

defmodule AshTherapy.Therapy.Message do
  use Ash.Resource,
    data_layer: AshPostgres.DataLayer

  postgres do
    # Database table and Repo configuration
  end

  attributes do
    # Resource attributes
  end

  relationships do
    # Resource associations
  end

  actions do
    # Resource actions
  end
end

Now, instead of showing full files for Conversation and Message, let’s highlight just the most interesting snippets from the AshTherapy project to understand each part.

Attributes

Here, we describe what information a conversation holds using the attribute DSL. This might seem familiar to Ecto, where we use the field macro. However, the attribute macro is more powerful and offers a lot more options.

attributes do
  uuid_primary_key :id

  attribute :goal, :string do
    allow_nil? false
    description "User’s goal or motive for starting this session."
  end

  attribute :status, :atom do
    constraints one_of: [:active, :closed]
    default :active
    description "Lifecycle state of the session."
  end

  attribute :started_at, :utc_datetime do
    default &DateTime.utc_now/0
  end
end

Declaring Relationships

Every conversation belongs to a user and contains messages. That’s expressed like this in the Conversation resource:

relationships do
  belongs_to :user, AshTherapy.Therapy.User, allow_nil?: true
  has_many :messages, AshTherapy.Therapy.Message
end

This tells Ash how our resources connect — it automatically handles loading, filtering, and association behaviors internally.

Declaring Business Actions

Now comes the heart of the resource: actions. Actions tell Ash what operations can be performed — creation, updates, reads, etc. Instead of writing explicit Elixir functions, we configure behavior through macros.

actions do
  defaults [:read]

  create :start_conversation do
    accept [:goal, :user_id]
    change set_attribute(:started_at, &DateTime.utc_now/0)
  end

  read :get_conversation_with_messages do
    argument :id, :uuid, allow_nil?: false
    filter expr(id == ^arg(:id))
    prepare build(load: [:messages])
  end
end

Let’s break down the start_conversation action:

It’s a create action for the Conversation resource.
It accepts only two inputs — goal and user_id.
It automatically sets started_at to the current time.
The status is already handled by its default value (:active).

If we compare this to a traditional Phoenix and Ecto setup, we’d have to:

Define a schema.
Write a changeset.
Write a function like create_conversation inside a context.
Call Repo.insert(changeset) manually.

With Ash, all of that is replaced by a simple, expressive declaration.

Reading Data with Built-in Preloading

The get_conversation_with_messages action shows another elegant pattern:

Fetch this conversation by ID and preload all related messages.

read :get_conversation_with_messages do
  argument :id, :uuid, allow_nil?: false
  filter expr(id == ^arg(:id))
  prepare build(load: [:messages])
end

No need to define query joins or preload manually. Ash interprets the relationships we declared earlier and does the heavy lifting.

In short:

We define what our business entities look like.
We describe how they relate.
We declare what can be done with them.

Ash then turns those configurations into complete, consistent, and database-backed operations — no repetitive boilerplate required.

Now that you’ve seen what Ash does at a high level, let’s take a deeper look at the philosophy that makes it so effective.

Declare Once, Derive the Rest

You might have heard the variation on Ash’s tagline:

Declare once, derive the rest.

This is not just a catchy phrase — it captures the very heart of what makes Ash powerful.

Let’s unpack that a bit.

If we look at an Ecto schema, it already feels declarative. We describe the shape of our data, the fields, their types — and that’s a great starting point. Conceptually, Ecto begins in the right place: it encourages us to describe what data looks like instead of writing procedural code.

But here’s where Ecto stops — and where Ash continues the journey.

In Ecto, that declarative schema doesn’t get reused elsewhere. After defining a schema, we still have to:

Write migrations manually, repeating much of the same information in SQL-like form.
Write context functions that perform create, read, or update operations.
Write controllers or APIs to expose those actions externally.

All of that is work we do again and again — even though the intent was already declared in the schema.

Frameworks like Rails or Django solved this years ago by reusing their models to drive migrations, validations, and APIs automatically. Ash brings that level of cohesion — and more — into the Elixir ecosystem.

Here’s how Ash builds on Ecto’s declarative foundation and takes it several steps further:

Once a resource is defined, Ash can generate migrations automatically. No need to manually write create table(...) or update migrations when attributes change. Even renames, removals, and relationship updates can be derived automatically.
The same resource definition also powers API generation. We can expose JSON:API, GraphQL, or RPC (like TypeScript remote procedure calls) in a single line of configuration — no need to write controllers or context functions.
All of this information — attributes, actions, relationships — stays consistent across the system because it comes from one single source of truth: the resource definition itself.

This is what is meant by “declare once, derive the rest”. We define our business logic in one place, and Ash intelligently derives everything else — migrations, APIs, validations, documentation — directly from that single declaration.

Quick Comparison: Ecto vs Ash

Here's a quick look at Ecto and Ash's features side-by-side:

Feature	Ecto	Ash
Schema Definition	Declarative	Declarative
Migrations	Manual	Auto-generated
Context Functions	Manual	Derived from Actions
API Layer	Handwritten	Auto-generated
Reuse	Limited	System-wide reuse

Ash turns your domain model into the single source of truth across all layers.

Playing with Ash in IEx

Let’s now play with what we’ve built and see Ash in action.

If you haven’t already cloned the repository, do it now and set up your project:

git clone https://github.com/devcarrots/ash_therapy.git
cd ash_therapy
mix deps.get
mix ash.codegen
mix ash.migrate
iex -S mix

Understanding `Ash.create` and `Ash.read`

So far, we have seen how to declare actions, but we haven't executed them yet. Ash provides simple functions to interact with these resource actions:

Ash.create/3 (or Ash.create!/3) executes a create action on a resource. You provide the resource module, the action name, and the data map for that action.
Ash.read/3 (or Ash.read!/3) executes a read action, usually returning filtered data.

Create a User

Before we can start a conversation, we need a user. I have already defined a register_user action in our User resource. It’s a simple create action that accepts a user’s name and stores a new record, similar in structure to the start_conversation action we explored earlier.

iex> user =
  Ash.create!(
    AshTherapy.Therapy.User,
    :register_user,
    %{name: "Nisha"}
  )

This executes the register_user action on the User resource and stores a record in the database. The returned struct includes an auto-generated UUID and timestamps.

Start a Conversation

Next, let’s start a conversation for this user. A conversation needs a goal (the user’s motive) and the user_id.

iex> conversation =
  Ash.create!(
    AshTherapy.Therapy.Conversation,
    :start_conversation,
    %{goal: "I want to handle work stress", user_id: user.id}
  )

This calls the start_conversation action, automatically setting the started_at timestamp. The status defaults to :active.

Send a Message

Now that we have a conversation set up, let’s send the first message. I have defined a send_message action in the message resource to take care of that. It's an action that can both send messages as a :user and also as the :ai agent (by setting the right :role attribute), as shown below.

iex> Ash.create!(
  AshTherapy.Therapy.Message,
  :send_message,
  %{
    content: "I feel anxious every Monday morning.",
    conversation_id: conversation.id
    # The default value of attribute `:role` is `:user`. So the below line is not needed.
    # role: :user
  }
)

The send_message action defaults the role to :user, sets the current timestamp, and saves the message.

To simulate an AI reply, we can reuse the same action, but explicitly set the role:

iex> Ash.create!(
  AshTherapy.Therapy.Message,
  :send_message,
  %{
    content: "That sounds difficult. What usually triggers this feeling?",
    conversation_id: conversation.id,
    # We set the `:role` to `:ai` when we want to override the default.
    role: :ai
  }
)

This reuse of the same action demonstrates how defaults and configuration keep Ash concise and flexible.

Fetch the Conversation Messages

Now let’s see the conversation and all the messages.

Ash.read!(
  AshTherapy.Therapy.Conversation,
  :get_conversation_with_messages,
  %{id: conversation.id}
)

The get_conversation_with_messages action loads the conversation and preloads all associated messages in one step. You’ll see a neat struct showing the conversation goal, status, and a list of messages (both from the user and AI).

And that's it!

Wrapping Up

In this post, we:

Explored what Ash is and how it complements Phoenix
Built three foundational resources: User, Conversation, and Message
Introduced attributes, relationships, and actions
Learned how Ash embodies the principle “declare once, derive the rest”
Tried out our domain interactively in IEx by calling Ash actions as if they were small, reusable business commands.

Happy Ash-ing! 👋

AppSignal’s Top 5 Elixir Posts in 2025

Roy Tomeij — 2025-12-17T00:00:00+00:00

With the year wrapping up, we’re excited to reveal the five Elixir articles you read the most in 2025.

Season's Greetings ❆ ⛄

Have a lovely holiday break. We'll be back with more Elixir posts when the new year begins!

If you haven’t already, don’t forget to subscribe to our Elixir Alchemy newsletter, so you never miss an upcoming post.

Build an Elixir App with Cowboy

Roy Tomeij — 2025-11-25T00:00:00+00:00

Cowboy is a small, fast, and modern HTTP server for Erlang/OTP. It particularly shines when handling multiple concurrent connections with minimal overhead. This makes it a perfect choice for building lightweight, real-time streaming services.

In this article, we’ll build a tiny real-time text pub/sub server using Cowboy. Our service supports HTTP-based publishing, allowing clients to subscribe using either WebSockets or Server-Sent Events (SSE). Think of it as a minimal message bus over the web, great for prototypes, internal dashboards, collaborative editors, or IoT telemetry applications.

Not only will you learn how to set up and use Cowboy, but you’ll also gain insights into designing a simple yet effective pub/sub architecture. By the end of this article, you’ll have a solid foundation to build upon for more complex real-time applications.

Why Cowboy for Elixir?

Phoenix is the go-to web framework in Elixir, but sometimes you don’t need the full stack. Cowboy shines when:

You want tight control over connections and protocols
You’re building a focused service (e.g., streaming telemetry or real-time feeds)
You want to avoid framework overhead and keep things lean

Cowboy’s design is close to the metal and as barebones as it gets. It doesn’t come with routing helpers or request processing pipelines—but that’s a feature, not a bug. You get raw access to HTTP and WebSocket layers, with no abstraction in your way. Perfect for protocol gateways, edge servers, or streaming APIs.

In contrast, Phoenix is fantastic for full-featured web apps with rich templating, channels, and LiveView. However, if your application's main responsibility is pushing data to clients with minimal latency, Cowboy’s simplicity can improve performance and reduce resource usage.

As mentioned in the introduction, we will build a simple text pub/sub server that supports publishing and subscribing via HTTP, WebSockets, and Server-Sent Events (SSE), which is a great fit for Cowboy’s strengths.

Project Setup and Bootstrapping Cowboy

Let's get started by creating a new Mix project and adding Cowboy as a dependency.

mix new text_bus --sup
cd text_bus

This will create a new Elixir project with a supervision tree. Next, open mix.exs and add Cowboy and Jason (for JSON encoding) to the dependencies:

defp deps do
 [
 {:cowboy, "~> 2.10"},
 {:jason, "~> 1.4"}
 ]
end

Fetch the dependencies:

mix deps.get

With our dependencies in place, we can set up Cowboy to start an HTTP listener on port 4000. The first step is to create a module to start the Cowboy server. Create a new file at lib/text_bus/http_server.ex:

defmodule TextBus.HTTPServer do
  use GenServer

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  def init(_opts) do
    dispatch =
      :cowboy_router.compile([
 {:_, [
 {"/health", TextBus.Handlers.Health, []}
 ]}
 ])

 {:ok, _} =
      :cowboy.start_clear(
        :http,
 [port: 4000],
 %{env: %{dispatch: dispatch}}
 )

 {:ok, %{}}
  end
end

While this code might look a bit complex, here’s a breakdown:

We define a TextBus.HTTPServer module that uses GenServer to manage the Cowboy server lifecycle.
In the init/1 function, we set up a simple router with a health check endpoint at /health.
We start the Cowboy server on port 4000, passing in our routing configuration.
Finally, we return the initial server state, an empty map.

Next, we need to make sure the application starts the Cowboy server. Open lib/text_bus/application.ex and modify the start/2 function to include our TextBus.HTTPServer module:

defmodule TextBus.Application do
  use Application

  def start(_type, _args) do
    children = [
      TextBus.HTTPServer
 ]

    Supervisor.start_link(children, strategy: :one_for_one, name: TextBus.Supervisor)
  end
end

Now let's create the health check handler in lib/text_bus/handlers/health.ex:

defmodule TextBus.Handlers.Health do
  @behaviour :cowboy_handler

  def init(req, state) do
 {:ok, req} =
      :cowboy_req.reply(200, %{"content-type" => "text/plain"}, "OK", req)

 {:ok, req, state}
  end
end

This handler returns a 200 OK response with the text "OK" as our health check. We can make sure our Cowboy scaffold is working by starting the application:

mix run --no-halt

And then using curl to hit the health endpoint:

curl -i http://localhost:4000/health

If everything is set up correctly, you should see a 200 OK response with the text "OK". Something like:

❯ curl -i http://localhost:4000/health
HTTP/1.1 200 OK
content-length: 2
content-type: text/plain
date: Thu, 18 Sep 2025 13:08:21 GMT
server: Cowboy

OK⏎

Before we start the implementation, let's quickly outline what a pub/sub server does:

Clients can publish messages to a topic via HTTP POST requests.
Clients can subscribe to topics via WebSockets or Server-Sent Events (SSE) to receive real-time updates.
The server maintains a buffer of recent messages for each topic to allow new subscribers to catch up.

Now, let's move on to the implementation.

Topic Model, Publish Flow, and Subscriptions

Let's define a Topic module to manage individual topics. Each topic will be a GenServer that maintains a ring buffer of messages and tracks subscribers. Create a new file at lib/text_bus/topic.ex:

defmodule TextBus.Topic do
  use GenServer

  @buffer_size 100

  # Client API
  def start_link(name), do: GenServer.start_link(__MODULE__, name, name: via(name))
  def publish(topic, msg), do: GenServer.cast(via(topic), {:publish, msg})
  def subscribe(topic, pid), do: GenServer.cast(via(topic), {:subscribe, pid})
  def unsubscribe(topic, pid), do: GenServer.cast(via(topic), {:unsubscribe, pid})
  def replay(topic, n), do: GenServer.call(via(topic), {:replay, n})

  defp via(name), do: {:via, Registry, {TextBus.TopicRegistry, name}}

  ## Server

  def init(name) do
    # Unnamed table: first arg can be any atom; result is a tid we keep in state.
    table = :ets.new(:topic_messages, [:ordered_set, :protected])
 {:ok, %{name: name, seq: 0, table: table, subs: MapSet.new()}}
  end

  def handle_cast({:publish, msg}, state) do
    seq = state.seq + 1
    :ets.insert(state.table, {seq, msg})
    Enum.each(state.subs, fn pid -> send(pid, {:message, state.name, seq, msg}) end)
 {:noreply, %{state | seq: seq}}
  end

  def handle_cast({:subscribe, pid}, state),
    do: {:noreply, %{state | subs: MapSet.put(state.subs, pid)}}

  def handle_cast({:unsubscribe, pid}, state),
    do: {:noreply, %{state | subs: MapSet.delete(state.subs, pid)}}

  def handle_call({:replay, n}, _from, state) do
    msgs =
      :ets.tab2list(state.table)
      |> Enum.sort_by(&elem(&1, 0))
      |> Enum.take(-n)

 {:reply, msgs, state}
  end
end

This Topic module does the following:

It uses GenServer to manage the state of each topic.
It maintains an ETS table as a ring buffer to store messages, allowing efficient storage and retrieval.
It tracks subscribers using a MapSet to avoid duplicates.
It provides functions to publish messages, subscribe/unsubscribe clients, and replay recent messages.

We will also need to set up a registry for our topics. Create a new file at lib/text_bus/topic_sup.ex:

defmodule TextBus.TopicSup do
  use DynamicSupervisor

  def start_link(_), do: DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__)

  def init(:ok), do: DynamicSupervisor.init(strategy: :one_for_one)

  def start_topic(name) do
    child_spec = {TextBus.Topic, name}
    DynamicSupervisor.start_child(__MODULE__, child_spec)
  end
end

Now, let's add a registry and the topic supervisor to our application supervision tree. Update lib/text_bus/application.ex:

defmodule TextBus.Application do
  use Application

  def start(_type, _args) do
    children = [
 {Registry, keys: :unique, name: TextBus.TopicRegistry},
      TextBus.TopicSup,
      TextBus.HTTPServer
 ]

    Supervisor.start_link(children, strategy: :one_for_one, name: TextBus.Supervisor)
  end
end

The {Registry, keys: :unique, name: TextBus.TopicRegistry}, line sets up an in-memory registry to keep track of our topics. Keep in mind that this is just for our tutorial; in production, you will want to use something with more persistence.

Now that we have a working topic model, we can implement the publish endpoint. Create a new handler at lib/text_bus/handlers/publish.ex:

defmodule TextBus.Handlers.Publish do
  @behaviour :cowboy_handler

  def init(req, state) do
    topic = :cowboy_req.binding(:topic, req)
 {:ok, body, req} = :cowboy_req.read_body(req)

    case Jason.decode(body) do
 {:ok, %{"message" => msg}} ->
        ensure_topic(topic)
        TextBus.Topic.publish(topic, msg)

        req =
          :cowboy_req.stream_reply(202, %{"content-type" => "application/json"}, req)

        :ok = :cowboy_req.stream_body(~s({"status":"accepted"}), :fin, req)
 {:ok, req, state}

      _ ->
 {:ok, req} = :cowboy_req.reply(400, %{}, "invalid", req)
 {:ok, req, state}
    end
  end

  defp ensure_topic(name) do
    case Registry.lookup(TextBus.TopicRegistry, name) do
 [] -> TextBus.TopicSup.start_topic(name)
      _ -> :ok
    end
  end
end

Next, add the publish route to our Cowboy router in lib/text_bus/http_server.ex:

dispatch = :cowboy_router.compile([
 {:_, [
 {"/health", TextBus.Handlers.Health, []},
 {"/publish/:topic", TextBus.Handlers.Publish, []},
 ]}
])

The next component is the SSE endpoint. This endpoint allows clients to subscribe to topics and receive real-time updates. SSE is dead simple: send "content-type" => "text/event-stream", keep the connection open, and push lines like data: hello.

Create a new handler at lib/text_bus/handlers/sse.ex:

defmodule TextBus.Handlers.SSE do
  @behaviour :cowboy_handler

  def init(req, state) do
    topic = :cowboy_req.binding(:topic, req)
    ensure_topic(topic)

    TextBus.Topic.subscribe(topic, self())

    req =
      :cowboy_req.stream_reply(200, %{
        "content-type" => "text/event-stream",
        "cache-control" => "no-cache",
        "connection" => "keep-alive"
 }, req)

    loop(req, topic)
 {:ok, req, state}
  end

  defp loop(req, topic) do
    receive do
 {:message, ^topic, seq, msg} ->
        frame = "id: #{seq}\ndata: #{msg}\n\n"
        :ok = :cowboy_req.stream_body(frame, :nofin, req)
        loop(req, topic)

 {:cowboy_req, :terminate} ->
        TextBus.Topic.unsubscribe(topic, self())
        :ok
    end
  end

  defp ensure_topic(name) do
    case Registry.lookup(TextBus.TopicRegistry, name) do
 [] -> TextBus.TopicSup.start_topic(name)
      _ -> :ok
    end
  end
end

Our next handler will be for WebSocket subscriptions. WebSockets provide a full-duplex communication channel over a single TCP connection, making them ideal for real-time applications.

Create a new handler at lib/text_bus/handlers/websocket.ex:

defmodule TextBus.Handlers.WS do
  @behaviour :cowboy_websocket

  def init(req, state) do
    topic = :cowboy_req.binding(:topic, req)
    ensure_topic(topic)
 {:cowboy_websocket, req, %{topic: topic}}
  end

  def websocket_init(state) do
    TextBus.Topic.subscribe(state.topic, self())
 {:ok, state}
  end

  def websocket_handle({:text, msg}, state) do
    TextBus.Topic.publish(state.topic, msg)
 {:ok, state}
  end

  def websocket_info({:message, _topic, seq, msg}, state) do
 {:reply, {:text, Jason.encode!(%{id: seq, data: msg})}, state}
  end

  def terminate(_reason, _req, state) do
    TextBus.Topic.unsubscribe(state.topic, self())
    :ok
  end

  defp ensure_topic(name) do
    case Registry.lookup(TextBus.TopicRegistry, name) do
 [] -> TextBus.TopicSup.start_topic(name)
      _ -> :ok
    end
  end
end

Finally, add the WebSocket route to our Cowboy router in lib/text_bus/http_server.ex:

dispatch = :cowboy_router.compile([
 {:_, [
 {"/health", TextBus.Handlers.Health, []},
 {"/publish/:topic", TextBus.Handlers.Publish, []},
 {"/sse/:topic", TextBus.Handlers.SSE, []},
 {"/ws/:topic", TextBus.Handlers.WS, []}
 ]}
])

Let's test our progress so far by starting the application again:

mix run --no-halt

Start by subscribing to a topic:

curl -N http://localhost:4000/sse/demo

And then in another terminal, using curl to publish a message:

curl -X POST -d '{"message":"Hello, World!"}' http://localhost:4000/publish/demo

If everything is working correctly, you should see the message appear in the SSE subscriber terminal.

curl -N http://localhost:4000/sse/demo
id: 1
data: Hello, World!

Replay, Stats, and Testing

Alright, although we technically have a working pub/sub server, it's only fully real-time. This means that if you are connected when a message is published, you will receive it. If you connect after a message is published, though, you will miss it. To solve this, we will add a replay endpoint that allows clients to request the last N messages for a topic.

Previously, we ensured that messages were stored in ETS (see TextBus.Topic). Now we will create a new handler to expose a replay endpoint. Create a new file at lib/text_bus/handlers/replay.ex:

defmodule TextBus.Handlers.Replay do
  @behaviour :cowboy_handler

  def init(req, state) do
    topic = :cowboy_req.binding(:topic, req)
    n = parse_n(:cowboy_req.qs_val("n", req, "10"))

    msgs =
      case Registry.lookup(TextBus.TopicRegistry, topic) do
 [] -> []
        _ -> TextBus.Topic.replay(topic, n)
      end

    body = Jason.encode!(Enum.map(msgs, fn {id, msg} -> %{id: id, data: msg} end))
 {:ok, req} = :cowboy_req.reply(200, %{"content-type" => "application/json"}, body, req)
 {:ok, req, state}
  end

  defp parse_n(val) do
    case Integer.parse(val) do
 {n, _} when n > 0 -> n
      _ -> 10
    end
  end
end

Don't forget to add the replay route to our Cowboy router in lib/text_bus/http_server.ex:

{"/replay/:topic", TextBus.Handlers.Replay, []}

Like we did before, we can test our progress so far by starting the application again and using curl:

# Start SSE subscription
curl -N http://localhost:4000/sse/demo

# In another shell, publish a few messages
curl -XPOST http://localhost:4000/publish/demo \
 -H "content-type: application/json" \
  -d '{"message":"hello"}'

curl -XPOST http://localhost:4000/publish/demo \
 -H "content-type: application/json" \
  -d '{"message":"world"}'

# Replay last 2 messages
curl "http://localhost:4000/replay/demo?n=2"
# => [{"id":1,"data":"hello"},{"id":2,"data":"world"}]

The next and final component that we will add is a stats endpoint. This endpoint will provide some basic introspection into our server, such as the number of topics, subscribers, and messages in buffers. Create a new handler at lib/text_bus/handlers/stats.ex:

defmodule TextBus.Handlers.Stats do
  @behaviour :cowboy_handler

  def init(req, state) do
    topics =
      Registry.select(TextBus.TopicRegistry, [{{:"$1", :_, :_}, [], [:"$1"]}])

 stats =
  Enum.map(topics, fn topic ->
        [{pid, _}] = Registry.lookup(TextBus.TopicRegistry, topic)
 %{topic: topic, subscribers: subscriber_count(pid)}
 end)

 body = Jason.encode!(%{topics: stats, total_topics: length(stats)})
 {:ok, req} = :cowboy_req.reply(200, %{"content-type" => "application/json"}, body, req)
 {:ok, req, state}
 end

 defp subscriber_count(pid) do
 # Reach into process state (simplest for demo purposes)
 case :sys.get_state(pid) do
 %{subs: subs} -> MapSet.size(subs)
 _ -> 0
 end
 end
end

Let's add the stats route to our Cowboy router in lib/text_bus/http_server.ex:

{"/stats", TextBus.Handlers.Stats, []}

Now let's test our progress so far by starting the application again and using curl:

# Start SSE subscription
curl -N http://localhost:4000/sse/demo

# In another shell, publish a few messages
curl -XPOST http://localhost:4000/publish/demo \
 -H "content-type: application/json" \
  -d '{"message":"hello"}'

curl -XPOST http://localhost:4000/publish/demo \
 -H "content-type: application/json" \
  -d '{"message":"world"}'

# Replay last 2 messages
curl "http://localhost:4000/replay/demo?n=2"
# => [{"id":1,"data":"hello"},{"id":2,"data":"world"}]

# Stats endpoint
curl http://localhost:4000/stats
# => {"status":"accepted"}{"status":"accepted"}[{"data":"hello","id":1},{"data":"world","id":2}]{"topics":[{"topic":"demo","subscribers":1}],"total_topics":1}

When to Use Cowboy vs. Phoenix for Elixir

At this point, we have a basic but fully functional pub/sub server built with Cowboy. You may wonder when to choose Cowboy over Phoenix for your projects.

Cowboy vs. Phoenix is not an either/or decision. They serve different purposes:

Cowboy Shines When…

You need full control over the protocol. Cowboy gives you raw access to HTTP, WebSockets, and even HTTP/2 without any framework abstractions in your way.
You’re building focused services. Things like telemetry pipelines, IoT data ingestion services, internal dashboards, or custom protocol gateways often just need a way to push data around quickly, not templating engines or an ORM.
You want to minimize overhead. Cowboy itself is small. No view layers, no router DSL, no middleware stack. That means fewer moving parts, less memory, and less latency.
You care about lightweight deployment. For sidecars, edge servers, or single-purpose services, starting from Cowboy keeps the footprint minimal.

Phoenix Makes More Sense When…

You’re building a full web application. If you need HTML templates, session management, static file serving, and a router DSL, Phoenix saves you a mountain of boilerplate.
You need higher-level abstractions. Phoenix Channels, Presence, and LiveView remove the need to roll your own pub/sub, replay, and state management.
You’re working in a team. Phoenix has a common structure, clear conventions, and rich documentation, which makes onboarding new developers easier.
You need built-in scaling tools. Phoenix PubSub is cluster-aware out of the box, so you don’t have to hand-roll distributed registries or message propagation.

Nothing illustrates this point better than the fact that Phoenix uses Cowboy under the hood. Here is how that works:

Cowboy is the HTTP/WebSocket server. It handles sockets, parses requests, and manages connections.
Plug is a middleware abstraction (think “Rack” in Ruby or “Express” in Node). It defines a standard way to build request/response pipelines.
Phoenix builds on Plug, adding routing, controllers, views, channels, LiveView, etc. Plug itself runs inside Cowboy.

Having a better understanding of Cowboy’s strengths will serve you well if you are working on the Elixir ecosystem. Plenty of production systems use Cowboy under the hood, for example:

RabbitMQ management & Web STOMP plugin – RabbitMQ (written in Erlang) uses Cowboy for its HTTP APIs, the management console, and WebSocket/STOMP support.
Elixir Language Server (ElixirLS) – the LSP server for editors like VS Code runs Cowboy directly to expose JSON-RPC endpoints over HTTP.
VerneMQ – a high-performance MQTT broker built in Erlang; its HTTP status and plugin endpoints are served through Cowboy.

Wrapping Up

To recap, we built a lightweight real-time pub/sub server directly on top of Cowboy. Along the way, you learned how to:

Set up Cowboy as a supervised OTP process and wire it into a simple Elixir app.
Define custom handlers for HTTP, SSE, and WebSocket endpoints.
Model topics as processes with an ETS-backed buffer for replay.
Expose introspection endpoints like /replay and /stats for debugging and observability.
Test the system end-to-end using curl, wscat, and a browser-based SSE client.

The result is a minimal but functional message bus: publishers post messages to a topic, subscribers receive them instantly, and operators can replay history or check system health. It’s a great foundation for prototypes, internal dashboards, IoT telemetry, or anywhere you need real-time streaming without the weight of a full framework.

Happy coding!

Debugging in Elixir with Observer

Roy Tomeij — 2025-11-04T00:00:00+00:00

Erlang's Observer is often discussed in passing and regarded as a curiosity during Elixir courses. However, Observer provides many powerful tools for monitoring and debugging your application, both in development and production.

Together, we will learn how to access the Observer GUI and debug a project that leaks memory, both locally and through a remote node. We will set up process tracing and track garbage collections to find the offending code in our sample project.

Let's get started!

Set Up: Verifying Observer in Your Elixir Installation

Start by opening an IEx session:

iex

Now try to start the Observer GUI:

iex> :observer.start()

The return value :ok should appear and a new window should open.

If you get an error message about a missing :observer module, your Erlang installation might have been compiled without the :observer or :wx modules, or WxWidgets might be missing from your system. Refer to the instructions of your toolchain manager of choice to follow along.

The Observer GUI in Erlang for Elixir

The GUI has 9 separate tabs:

System shows an overview of your system's capabilities, memory and CPU usage, and runtime statistics.
Load Charts displays live memory, scheduler utilization, and I/O statistics.
Memory Allocators shows the Erlang runtime's memory carriers usage.

Applications shows the different running applications and their processes' supervision trees.

Processes lists processes, the number of reductions, memory usage, and mailbox sizes.

Ports and Sockets, respectively, list open Erlang ports and sockets and their owner processes.
The Table Viewer tab lists viewable ETS tables. You can use the View > View Unreadable Tables OS menu to show more tables than the default state displays.

Trace Overview, where we can set up tracing and view traces for offending processes, which we will use extensively in the next steps of this article.

At this point, if you have already used the Phoenix Framework and the LiveDashboard package, you might notice that there is some intersection between what Observer and LiveDashboard provide. However, LiveDashboard is more tailored to monitoring web applications.

I encourage you to check out these various tools before continuing. As an example, try selecting applications to see their supervision tree, or inspect the contents of ETS tables.

We will now proceed to set up a project with a faulty process and debug it using Observer.

Project Setup: Elixir App with Leaking Memory

We will create a simple Elixir application with a process that periodically leaks memory. This problem can occur in real applications if multiple processes get hold of large binaries, because the runtime shares large binaries stored in a dedicated heap instead of copying them.

Creating the Project

In your terminal, create the project with Mix, including a supervision tree, and change directory to this new folder:

mix new demo_app --sup
cd demo_app

Add the Offending Process Module

Create a new lib/demo_app/offending_process.ex file:

defmodule DemoApp.OffendingProcess do
  use GenServer
  require Logger

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_opts) do
    schedule_work()
    {:ok, %{binaries: []}}
  end

  def handle_info(:work, state) do
    large_binary = :crypto.strong_rand_bytes(512_000)

    new_state = %{state | binaries: [large_binary | state.binaries]}

    IO.puts("OffendingProcess created binary.")

    schedule_work()

    {:noreply, new_state}
  end

  defp schedule_work do
    Process.send_after(self(), :work, 2_000)
  end
end

Update the Application Supervisor

Edit lib/demo_app/application.ex:

defmodule DemoApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Add the OffendingProcess to the supervision tree
      DemoApp.OffendingProcess
    ]

    opts = [strategy: :one_for_one, name: DemoApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Add the Observer Backend

Edit mix.exs to add the :runtime_tools application:

def application do
  [
    extra_applications: [:logger, :runtime_tools],
    mod: {DemoApp.Application, []}
  ]
end

Let's start the application in IEx with a named node, so we can connect to it from another node running Observer.

iex --name demo@127.0.0.1 --cookie democookie -S mix

You should see messages printed every 2 seconds.

OffendingProcess created binary.

Connecting Observer from a Separate Node

In a new terminal, start a second hidden IEx node.

iex --hidden --name observer@127.0.0.1 --cookie democookie

Then in the IEx session, connect to the first node and start Observer:

iex> Node.connect(:"demo@127.0.0.1")
true
iex> :observer.start(:"demo@127.0.0.1")
:ok

If you navigate to the Processes tab, you should see the OffendingProcess's memory usage slowly increase as the number of references to large binaries it holds grows.

In the System tab, you can see overall memory usage and binary-specific memory usage increase more quickly.

This two-node setup means we can avoid interfering with resource usage measurements on the first node, by not running the Observer app directly on it.

Debugging a Remote System with Observer

Let's set up tracing for our offending process from the Observer GUI controlled by our hidden node. Go to the Processes tab, right-click on Elixir.DemoApp.OffendingProcess, and select "Trace selected processes".

In the process options, select the following:

Trace function call
Trace arity instead of arguments
Trace garbage collections

Move to the Trace Overview tab and click on the process now listed in the rightmost part of the GUI.

Click on the "Add Trace Pattern" button at the bottom of the GUI. In the pop-up that opens, select the module that we are looking to trace: Elixir.DemoApp.OffendingProcess.

In the second pop-up that opens, select functions to trace. We will select schedule_work/0.

The next pop-up asks us to select or write a Match Specification for our traces. If you are already familiar with :ets (the Erlang Term Storage application), the match specifications used to perform queries with :ets.select/2 are the same concept, despite having a slightly different grammar. Match specifications are detailed in the documentation of the Erlang Runtime System Application (ERTS).

For our purpose, we will select the pre-existing "Return Trace" match specification. Here, the return_trace function causes a return_from trace message to be sent when the selected function returns.

In the leftmost part of the window, click on "Start Trace".

A window will open where each schedule_work/0 call will log the entry and return from that function.

After a while, you will notice a different log: minor garbage collections triggered by the VM. In short, a garbage collection is an attempt to free memory by deleting data that cannot be referenced anymore.

In the screenshot, we can read information about the garbage collection that just occurred. All sizes are expressed in words.

old_heap_block_size is the size of the memory block storing the heap pre-collection
heap_block_size is the size of the memory block storing both the heap and the stack
mbuf_size is the size of the process's message buffers
recent_size is the size of the data that passed the previous garbage collection
stack_size is the size of the stack
old_heap_size is the size of block heap actually used, pre-collection
heap_size is the size of block heap actually used
bin_vheap_size is the size of binaries referenced from the process heap
bin_vheap_block_size is the size of binaries allowed in the process's virtual heap before triggering a garbage collection

The key that interests us the most here is bin_vheap_size, describing the space taken by binaries referenced in the process. We are talking about a virtual heap here because large binaries aren't actually stored in the process's heap, but in a special and global binary heap as an optimization. In writing this article, when I first started tracing, this key read 7_168_092 words. After a while, it grew to 44_096_000 words.

Indeed, going back to the "System" tab in the Observer GUI, I can see 372MiB taken by the binaries.

And this concludes our overview of the Observer GUI!

Going Further

As you can see, the Observer GUI's capabilities are quite extensive and it is difficult to provide an exhaustive tour here.

Here are a few things you can try yourself to dive a bit deeper:

Find the supervision tree of an application in the "Applications" tab, see its supervision strategies, and terminate a few of its children
Inspect the state of a long-running process. Can you see all of its internal state in your IEx session?
Trace specific functions through your application, then try to trace them without Observer, with :erlang.trace/3
Try to start etop, included with the Observer application, with erl -name etop -hidden -s etop -s erlang halt -output text -node demo@127.0.0.1 -setcookie democookie and see how this compares to the graphical Observer
Explore the trace tool builder ttb and crashdump_viewer, two other tools that come with the Observer application.

Wrapping Up

In this post, we took a quick tour of Erlang's Observer GUI. We first set up Observer for an Elixir application, then used an example app with a memory leak to explore some of Observer's capabilities.

We debugged a remote system, before finally suggesting some ways to dive deeper into exploring Observer's many functions.

Happy coding!

Monitor the Performance of Your Ecto for Elixir App with AppSignal

Roy Tomeij — 2025-10-28T00:00:00+00:00

In part one of this series, we learned how to implement batch updates and advanced inserts in Ecto to dramatically improve database performance.

But implementing these optimizations is only the first step. Ensuring they continue to work effectively in production requires professional monitoring and observability.

This guide will show you how to use AppSignal for Elixir to monitor your Ecto application's performance when dealing with batch data operations.

Prerequisites

An Elixir/Phoenix application. If you don't have one, fork the e-commerce Phoenix app we'll be using throughout this tutorial.
Elixir, Phoenix framework, and PostgreSQL installed on your local development environment.
An AppSignal account. Sign up for a free trial if you don't have one.
Basic experience working with Elixir and the Phoenix framework.

Next, let's learn how to set up AppSignal for Elixir in our example app.

Setting Up AppSignal for Elixir

Open up the app's mix.exs file and add the AppSignal for Elixir package:

# mix.exs
def deps do
  [
    ...
    {:appsignal, "~> 2.8"},
    {:appsignal_phoenix, "~> 2.0"}
  ]
end

Note: If you have a Phoenix application, make sure you also add the AppSignal for Phoenix package, which depends on the AppSignal package.

Then run mix deps.get to add the packages.

Next, run mix appsignal.install YOUR_PUSH_API_KEY to complete the installation. When you run this command, you will be prompted to configure how your app will be named in the AppSignal dashboard. You can also decide how the AppSignal for Elixir package will be integrated into your app, either through a configuration file (recommended), or an environment variable.

Assuming the installation goes smoothly, you should start seeing some default metrics on the AppSignal dashboard after a few minutes:

And that's all set up.

Understanding Ecto Performance Metrics

With AppSignal installed, we can now move on to learning about some key performance indicators that AppSignal for Elixir monitors by default, namely query execution time, throughput, and N+1 queries.

Query Execution Time and Throughput

Execution time tells you how long individual database operations take, while throughput indicates how many queries your application processes over time.

The AppSignal dashboard provides a convenient timeline visualization that breaks down request execution across different layers of your Phoenix app. You can see how a single request is distributed across the Phoenix endpoint and template, the router, and Ecto.

In the example shown in the screenshot above, the GET request to fetch products completed successfully, taking a total of 6ms, out of which Ecto takes up 2ms — around 33% of the total execution time.

When monitoring query time and throughput, these are some things to watch for:

Consistent execution times - You're looking for consistent execution times across similar queries. If you start to experience longer execution times (compared to what you have noted as the average), you may need to look at optimizing Ecto queries.
Declining throughput while execution time increases - This metric could point to a growing bottleneck in your app's Ecto queries.
Query run times - Let's say you have two Ecto queries. One query takes around 10ms to execute but is being called a thousand times in an hour, while the other query takes 100ms, but is called 5 times in an hour. In such a scenario, you might think that the query that takes up more time should be prioritized for optimization, but if you turn your attention to the other and cut query time by 20%, you'll probably impact your app's performance much more than by fixing the longer query.

N+1 Queries

N+1 queries occur when an initial query triggers additional queries for each returned record.

Using the e-commerce app as an example, consider the code snippet below:

# lib/shopflow/catalog.ex

defmodule Shopflow.Catalog do
  ...
  def list_products_with_n_plus_one do
    products = Repo.all(Product)  # 1 query for products

    Enum.map(products, fn product ->
      supplier = Repo.get(Supplier, product.supplier_id)  # N queries for suppliers
      %{product | supplier: supplier}
    end)
  end
  ...
end

Here, the list_products_with_n_plus_one function fetches all products, then makes additional database calls to fetch associated suppliers, a classic N+1 query (as you can see in the log outputs below):

[debug] Processing with ShopflowWeb.ProductController.index/2
  Parameters: %{}
  Pipelines: [:browser]
[debug] QUERY OK source="products" db=3.9ms queue=0.2ms idle=648.6ms
SELECT p0."id", p0."name", p0."description", p0."sku", p0."price", p0."is_active", p0."deleted_at", p0."category_id", p0."supplier_id", p0."inserted_at", p0."updated_at" FROM "products" AS p0 []
↳ Shopflow.Catalog.list_products_with_n_plus_one/0, at: lib/shopflow/catalog.ex:125
[debug] QUERY OK source="suppliers" db=1.3ms queue=0.1ms idle=659.2ms
SELECT s0."id", s0."name", s0."contact_email", s0."contact_phone", s0."is_active", s0."inserted_at", s0."updated_at" FROM "suppliers" AS s0 WHERE (s0."id" = $1) ["f861e835-6618-4613-bd74-56667be8c01c"]

The impact can be seen in the time Ecto takes to process the request — a huge 42% of the total time taken by the product listing request:

See the measured impact on this unoptimized query:

Next, let's improve the query by using Ecto.preload/3.

First, edit the function like so:

# lib/shopflow/catalog.ex

defmodule Shopflow.Catalog do
  ...
  def list_products do
    Product
    |> Repo.all()
    |> Repo.preload(:supplier)
  end
  ...
end

And on reloading the products list, we can see how Ecto.preload/3 cuts down the impact metric:

To a large extent, the AppSignal for Elixir integration will take care of monitoring most parts of Ecto. However, if we want to get deeper insights or coverage for scenarios that are unique to the application at hand, then we need to add custom instrumentation.

Beyond Basics — Monitoring Ecto for Elixir with AppSignal Custom Instrumentation

In part one, we learned how to insert and modify large datasets using Ecto's insert_all/3 and the Ecto.Multi module.

In this section, we'll take the examples we started with in part one and apply AppSignal for Elixir's custom instrumentation.

Monitoring Ecto Batch Inserts

In part one, we learned about the different parts that make up Ecto, like the schema, changeset, repo, and how they interact when working with batch operations.

For this example, we'll go further by learning how to insert bulk data into the database using CSV files. Then we'll explore how to instrument these batch data operations using AppSignal for Elixir's custom instrumentation.

Before diving into any code examples, it's important to get an overview of the CSV import process I've just mentioned.

The simplified CSV import will proceed as follows:

User clicks on a CSV import button on the frontend, which opens up a dialog for the CSV import to happen.
The request is sent to the router, then on to the controller, which then leverages a CsvImporter module to process the CSV and insert valid product records into the database.

Next, let's take a look at some of the parts involved in the process since this will help us understand where to leverage the custom instrumentation later on.

To start us off is the user interface:

<!-- lib/shopflow_web/controllers/product_html/index.html.heex -->
...

<button onclick="document.getElementById('import-modal').classList.remove('hidden')">
  Import CSV
</button>

<!-- Modal with file upload -->
<.simple_form for={%{}} action={~p"/products/import"} multipart={true}>
  <.input field={f[:csv_file]} type="file" accept=".csv" />
</.simple_form>
...

Next is the router:

# lib/shopflow_web/router.ex
...
post "/products/import", ProductController, :import
..

Then the controller:

# lib/shopflow_web/controllers/product_controller.ex
...
def import(conn, %{"csv_file" => upload}) do
  case validate_csv_file(upload) do
    {:ok, file_path} -> # Process the file
    {:error, reason} -> # Show error message
  end
end
...

And finally, the CSV import module, with a focus on the bit that does the database insert:

# lib/shopflow/catalog/csv_importer.ex
...
Repo.transaction(fn ->
  # Add timestamps and UUIDs
  products_for_insert = Enum.map(products_data, fn product_data ->
    product_data
    |> Map.put("id", Ecto.UUID.generate())
    |> Map.put("inserted_at", now)
    |> Map.put("updated_at", now)
  end)

  # Actual bulk insert
  {count, _} = Repo.insert_all(Product, products_for_insert)
  count
end)
...

When using Repo.insert_all/3, you should note that autogenerated attributes like record IDs, as well as inserted_at and updated_at, will not be automatically generated for you, so make sure you do this manually.

With that in place, you might be wondering why we even need custom instrumentation when the AppSignal integration already gives us excellent Ecto monitoring?

Why Use Custom Instrumentation for Your Elixir App?

For starters, take a look at one of the products' controller request samples, with a focus on Ecto's metrics (the products controller is where the CSV import happens):

And this one:

As you can see, even though AppSignal for Elixir covers basic Ecto metrics, to get specific data on the CSV import process would require setting up custom metrics.

Let's learn how to do that next.

Custom Instrumenting Ecto Batch Inserts

Custom instrumentation allows you to add detailed monitoring to specific parts of your Ecto operations that aren't automatically tracked by AppSignal for Elixir's default integration.

By adding custom instrumentation, you can track metrics like:

How long bulk operations take to complete.
Success and failure rates for batch processes.
Custom metrics related to your specific use case

And more.

For now, let's instrument the CSV import to measure the frequency of CSV import failures, a metric that might negatively impact user experience.

Here's a brief snippet of the instrumented import_csv/1 function with success/failure counters:

# lib/shopflow/catalog/csv_importer.ex
...

def import_csv(file_path) do
    result = with {:ok, csv_data} <- read_and_parse_csv(file_path),
         {:ok, headers} <- validate_headers(csv_data),
         {:ok, products_data} <- validate_and_transform_data(csv_data, headers) do
      insert_products(products_data)
    end

    # Track failures with AppSignal
    case result do
      {:error, _reason} ->
        Appsignal.increment_counter("csv_import.failure")
        Appsignal.add_distribution_value("csv_import.error_rate", 1)
      {:ok, _} ->
        Appsignal.increment_counter("csv_import.success")
        Appsignal.add_distribution_value("csv_import.error_rate", 0)
      _ ->
        :ok
    end

    result
  end
  ...

In this code example, we're using two AppSignal functions to collect custom metrics for our CSV import operation:

Appsignal.increment_counter("csv_import.success") - This increments a counter metric each time a CSV import completes successfully. Counters are perfect for tracking the total number of events over time, giving us visibility into import frequency and success rates.
Appsignal.add_distribution_value("csv_import.error_rate", 0) - This adds a value of 0 to a distribution metric when imports succeed. Distribution metrics help us track statistical data like averages, percentiles, and patterns. By recording 0 for successful imports (and 1 for failures), we can calculate error rates and monitor import reliability trends over time.

These custom metrics complement AppSignal's automatic Ecto instrumentation by providing business-specific insights that help us understand not just database performance, but also the success patterns of the CSV import operation.

The results are seen in the custom dashboard below:

Note: The process of adding custom dashboards is not included in this tutorial, but you can follow this guide to learn more.

Wrapping Up

In this guide, we explored how to effectively monitor Ecto performance using AppSignal for Elixir, building on the batch data techniques we learned in part one.

We covered the setup process, examined a few key performance metrics like query execution time, throughput, and N+1 queries, and learned how to implement custom instrumentation for scenarios that require deeper monitoring.

With the lessons covered in this tutorial, you now have the skills to identify and measure various performance bottlenecks in your Ecto applications.

Happy coding!

Batch Updates and Advanced Inserts in Ecto for Elixir

Roy Tomeij — 2025-10-07T00:00:00+00:00

When you build Elixir applications, you'll likely encounter scenarios where you need to handle large datasets efficiently. Whether you're importing user data from a CSV file, updating thousands of product prices, or synchronizing data from external APIs, performing operations one record at a time can quickly become a performance bottleneck.

In this two-part tutorial, we'll start by exploring Ecto's powerful batch update and advanced insert capabilities for handling bulk data operations. Then, in the second part, we'll learn how to integrate AppSignal for Elixir to build an observability layer on top of Ecto to catch errors and any performance issues that could arise.

Before we begin, you'll need a few things set up.

Prerequisites

Elixir, the Phoenix framework and PostgreSQL installed and running in your local development environment.
A Phoenix application. If you don't have one ready, fork the example app we'll be using throughout this tutorial.
Basic Elixir and Phoenix framework experience.

Introducing Ecto for Elixir

Ecto is Elixir's database toolkit that acts as a bridge between your application and relational databases like PostgreSQL. It provides type-safe query building, automated migrations, and data validation, making complex data operations simple and reliable.

Ecto's Architecture

At its core, Ecto is built around three fundamental components that work together to provide a complete database abstraction layer:

Schemas act as the structural foundation, defining how data maps to database tables without containing any business logic. Schemas establish field types, relationships, and table mappings.
Changesets form the business logic layer, handling validation, data transformation, and change tracking. They take raw input data and a schema struct, then produce a validated, trackable representation of what should change in the database.
The Repo translates Ecto operations into actual SQL queries, while managing database connections, transactions, and query execution.

Next, we'll take a look at how Ecto handles single database operations, the challenges that come with it, and why batch operations matter.

How Standard Database Operations Work in Ecto for Elixir

For individual database operations like inserts, updates, or deletes, Ecto validates your data through changesets before sending SQL commands to the database. This approach works well for single-record operations but can become a performance bottleneck when handling a large number of records.

Before we learn how standard operations work through an example, let's get an overview of the app we'll be using moving forward.

Introducing Our Example Phoenix App

Our example app is a Phoenix ecommerce app managing products, suppliers, orders, products, and inventory for multiple stores. The app's database is represented below:

Now that we have an idea of how our app is structured (at least on the database level), in the next section, we'll learn how to do a normal data insert using Ecto.

Standard Inserts with Ecto

A database insert operation creates a new record in a table. In Ecto, this process involves several steps: first, you create a changeset that validates and transforms your data according to your schema's rules, then Ecto converts this changeset into the appropriate SQL INSERT statement and executes it against the database.

Next, let's learn about the role of each Ecto component.

The Schema’s Role

We first start with the supplier schema:

# lib/shopflow/suppliers/supplier.ex

defmodule Shopflow.Suppliers.Supplier do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id

  schema "suppliers" do
    field :name, :string
    field :contact_email, :string
    field :contact_phone, :string
    field :is_active, :boolean, default: false

    timestamps(type: :utc_datetime)
  end

  @doc false
  def changeset(supplier, attrs) do
    supplier
    |> cast(attrs, [:name, :contact_email, :contact_phone, :is_active])
    |> validate_required([:name])
    |> validate_format(:contact_email, ~r/@/)
  end
end

This schema definition establishes several important aspects of our Supplier model:

Primary key configuration: Uses :binary_id with auto-generation, creating UUID-based primary keys instead of sequential integers for better distribution across systems.
Foreign key type: Sets :binary_id as the default foreign key type to maintain consistency with the primary key format.
Field definitions: Defines four main fields — name (string), contact_email (string), contact_phone (string), and is_active (boolean with false default).
Timestamps: Automatically adds inserted_at and updated_at fields using UTC datetime format.
Validation rules: The changeset function enforces that name is required and contact_email follows a basic email format pattern.
Data casting: Only allows specific fields to be modified through the cast/3 function, providing a controlled interface for data changes.

The Repo’s Role

Next, we have the repo:

# lib/shopflow/repo.ex

defmodule Shopflow.Repo do
  use Ecto.Repo,
    otp_app: :shopflow,
    adapter: Ecto.Adapters.Postgres
end

The repo serves as the database interface layer: the bridge between our Elixir code and the PostgreSQL database, handling all database operations for our schemas.

With these in place, we can now create a database record by performing an insert.

Creating a Record in the Database

With the application running, open another terminal and run iex -S mix to start an interactive Elixir shell. We'll use it to create a new supplier record with the commands below:

# 1. Alias the Repo and Supplier schema to have access them in the shell
alias Shopflow.Repo
alias Shopflow.Suppliers.Supplier

# 2. Define attributes for the new supplier
attrs = %{
  name: "Spaceship suppliers",          # Required field
  contact_email: "info@example.com",  # Validated by regex (~r/@/)
  contact_phone: "123-456-7890",      # Optional
  is_active: true                     # Default is false
}

# 3. Build a changeset to validate and prepare the data
changeset = Supplier.changeset(%Supplier{}, attrs)

# 4. Insert the changeset into the database using the Repo
Repo.insert(changeset)

If successful, you’ll see a map of the inserted supplier, including the generated id and timestamps:

{:ok,
 %Shopflow.Suppliers.Supplier{
   __meta__: #Ecto.Schema.Metadata<:loaded, "suppliers">,
   id: "f861e835-6618-4613-bd74-56667be8c01c",
   name: "Spaceship suppliers",
   contact_email: "info@example.com",
   contact_phone: "123-456-7890",
   is_active: true,
   inserted_at: ~U[2025-08-26 09:21:00Z],
   updated_at: ~U[2025-08-26 09:21:00Z]
 }}

While the single-record approach shown above works perfectly for individual operations, it becomes problematic when dealing with large datasets.

The Performance Problem and Why Batch Operations Matter

Each individual insert requires a separate database connection and network round-trip, as you can see when we created the supplier:

iex(5)> Repo.insert(changeset)
[debug] QUERY OK source="suppliers" db=6.2ms decode=1.5ms queue=0.8ms idle=1234.3ms
INSERT INTO "suppliers" ("name","contact_email","contact_phone","is_active","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) ["Spaceship suppliers", "info@example.com", "123-456-7890", true, ~U[2025-08-26 09:21:00Z], ~U[2025-08-26 09:21:00Z], "f861e835-6618-4613-bd74-56667be8c01c"]

To create 10,000 suppliers, you make 10,000 separate database calls, which is very inefficient. To make matters worse, you lose atomicity — if the 500th supplier fails to insert due to a validation error, the previous 499 records will have already been committed to the database, leaving your data in an inconsistent state.

This is where Ecto's batch operations become essential.

Batch Inserts in Ecto: Inserting Multiple Records Efficiently

Batch inserts in Ecto allow you to create multiple database records in a single, efficient operation using Repo.insert_all/3. Unlike individual inserts that require separate database round-trips for each record, batch inserts compile all your data into a single SQL statement that creates hundreds or thousands of records at once.

An Example: Bulk Inserting Supplier Data

In the previous example, we used Repo.insert/2 to insert a single supplier record into the database, with the changeset performing validations and other checks before insertion into the database.

For a bulk insert, we'll make use of Repo.insert_all/3 to insert a bunch of suppliers into the database.

Preparing Data for Batch Insertion

To begin with, prepare a relevant data source. In my case, I prepared a CSV with around 1,000 supplier records.

The thing to note when preparing data for batch insertion is to make sure the data is in the right format. Remember, batch inserts do not utilize a schema's changeset to help with any validations or data checks, which means data validation is up to you.

Using `Repo.insert_all/3` To Bulk Insert Data

Next, with the server running in one terminal, open another terminal and create a new iEx session with iex -S mix, then run the command below:

alias NimbleCSV.RFC4180, as: Parser
alias Shopflow.Repo

Note: Prior installation of NimbleCSV or any other CSV library is required to process CSV files.

First, we alias NimbleCSV and the repo to make them available in the iEx session. Then, we run the following commands:

#1: We get the location of the CSV file
csv_content = File.read!("path-to-csv-file")

#2: Parse the CSV file
parsed_rows = Parser.parse_string(csv_content)

#3: Use Repo.insert_all/3 to do the bulk insert
Repo.insert_all(
  "suppliers",
  Enum.map(parsed_rows, fn [n, e, p, a] ->
    %{
      id: Ecto.UUID.generate() |> Ecto.UUID.dump!(),
      name: n,
      contact_email: e,
      contact_phone: p,
      is_active: String.downcase(a) == "true",
      inserted_at: DateTime.utc_now(),
      updated_at: DateTime.utc_now()
    }
  end)
)

Here, each CSV row is converted into a database-ready map with generated UUIDs, string values are converted to appropriate data types, timestamps are added, and finally, all records are inserted into the suppliers table in a single transaction using Repo.insert_all/3.

The Challenges of Batch Inserts

While Repo.insert_all/3 provides excellent performance for bulk operations, it comes with several important limitations that developers need to understand.

Validation and Error Handling

Unlike individual inserts that leverage Ecto's changeset system, batch inserts bypass validation entirely, meaning data integrity checks must be handled manually before insertion.

For example, we know that database-level constraints like unique indexes can cause the entire batch to fail if even a single record violates them. A good strategy to deal with this is to use a combination of smaller batches and Ecto's transaction API, as in the example below:

batch_size = 100
suppliers_attrs
|> Enum.chunk_every(batch_size)
|> Enum.each(fn batch ->
  Repo.transaction(fn ->
    # Attempt to insert the batch
    Repo.insert_all("suppliers", batch)
  rescue
    Ecto.ConstraintError ->
      # Handle duplicate email (or other constraint) here
      IO.puts("Batch failed due to constraint violation. Retrying...")
  end)
end)

We first divide the suppliers_attrs into batches of 100 records each, then process each batch sequentially, with each batch being inserted into the database using Repo.insert_all/3 wrapped in a database transaction. This will rollback the transaction should an error occur (in this case, we try to rescue the error and handle it in some way).

Limited Association Handling

Repo.insert_all/3 cannot directly handle associations (for example, linking suppliers to products). In such cases, you must handle the association manually by first creating the parent records, then referencing the parent record IDs when creating the child records.

Next, let's switch gears to the subject of batch updates.

Batch Updates

Batch updates involve updating a number of records all at once. They introduce some unique challenges, including:

Interdependencies: Updates often affect associated records. For example, changing a supplier’s status requires updating linked products at the same time.
Partial failures: A single invalid change (for instance, violating a uniqueness constraint can derail the entire batch).
Concurrency risks: Competing updates to the same records can cause race conditions (for example, overlapping inventory adjustments).

So, what options do you have if you want to perform a stress-free batch update? One solution is to use Ecto.Multi.

Handling Batch Updates With `Ecto.Multi`

Ecto.Multi is a tool that lets you bundle multiple database operations (inserts, updates, and deletes) into a single chunk. Ecto.Multi solves the critical issues that come with bulk updates, namely:

Atomicity - Ensuring all steps succeed (or none do).
Dependency management - Making sure associations and dependencies are handled efficiently.
Error isolation - With Ecto.Multi, each step is named, which makes it very easy to pinpoint where a failure occurs.

An Example Using `Ecto.Multi` to Run a Batch Update

Let's say we need to deactivate a supplier and mark all their products as discontinued. How can we do this effectively?

Using Ecto.Multi, we could run code like this:

iex(7)>
alias Ecto.Multi
alias Shopflow.Suppliers.Supplier
alias Shopflow.Products.Product

# Target supplier ID to deactivate
supplier_id = "f861e835-6618-4613-bd74-56667be8c01c"

# Compose multi-step update
multi =
  Multi.new()
  # Step 1: Deactivate the supplier
  |> Multi.update(:deactivate_supplier,
    Supplier
    |> where([s], s.id == ^supplier_id)
    |> select([s], s),  # Fetch the supplier record
    set: [is_active: false]  # Set active flag to false
  )
  # Step 2: Mark all their products as inactive
  |> Multi.update_all(:discontinue_products,
    Product
    |> where([p], p.supplier_id == ^supplier_id),
    set: [is_active: false]  # Update all matching products
  )

# Execute the transaction
case Repo.transaction(multi) do
  {:ok, _results} ->
    IO.puts("Supplier and products updated successfully!")
  {:error, step, reason, _} ->
    IO.puts("Update failed at step '#{step}': #{inspect(reason)}")
end

Here, we use Ecto.Multi wrapped in a database transaction (so a rollback can happen, in case of errors) to create a multi-step update. The update starts with a step called :deactive_supplier, which finds the specific supplier we want. Then, the :discontinue_products step sets the is_active flag as false on any product belonging to the supplier.

And that's it for this first part!

Wrapping Up

In this tutorial, we learned how to handle bulk data operations efficiently using Ecto's powerful batch capabilities. We explored the performance limitations of single-record operations and saw how Repo.insert_all/3 can dramatically improve insertion performance for large datasets. We also found out about the challenges that come with batch operations, and how we can use Ecto.Multi to ensure data consistency when doing batch updates.

In part two of this series, we'll integrate AppSignal for Elixir into our app to build an observability layer that monitors these batch operations. This will help you catch performance bottlenecks and errors before they impact your production applications.

Until then, happy coding!

Advanced Debugging in Elixir with IO.inspect

Roy Tomeij — 2025-09-23T00:00:00+00:00

When writing Elixir, most developers quickly get familiar with IO.inspect as a quick way to see what's happening inside their code. But what many overlook is that IO.inspect is far more powerful than just a method that prints a variable to the console.

In fact, with the right options and placement, IO.inspect can become a precise, highly targeted debugging tool, one that doesn't interrupt your program flow and works seamlessly with Elixir's functional pipelines.

This post will walk you through both the fundamentals and advanced patterns for using IO.inspect effectively. By the end, you'll know how to control output formatting, label your prints for clarity, debug concurrent processes, and even integrate conditional or file-based inspection.

The Basics of `IO.inspect` for Elixir

IO.inspect is defined like this:

IO.inspect(item, opts \\ [])

item: Any Elixir value you want to inspect.
opts: Keyword list of options that controls how the term is printed, as defined in Inspect.Opts.

By default, it writes the inspected value to the standard output (:stdio) and then returns the term unchanged.

That last part is key. Because it returns the term, you can drop it anywhere in a function or pipeline without breaking the flow.

Example:

defmodule Demo do
  def run do
    IO.inspect(%{name: "Alice", age: 30})
  end
end

Prints:

%{age: 30, name: "Alice"}

Caveat: When `IO.inspect` Is the Last Call in a Function

Because IO.inspect returns its argument, if it's the last expression in your function, that inspected value becomes the function's return value.

That's fine if you intend to return it, but it can lead to subtle bugs if you were only printing it for debugging and expected a different return.

Take this example:

def fetch_user(id) do
  Repo.get(User, id)
  |> IO.inspect(label: "Fetched user")
end

Here, the function returns the user struct as normal, which is fine.

Now, consider:

def log_and_return do
  IO.inspect("Done!")
end

This returns "Done!" instead of, say, :ok.

If you want to print something but return a different value, make the return explicit:

def log_and_return do
  IO.inspect("Done!")
  :ok
end

Or if you're in a pipeline:

value
|> IO.inspect(label: "Debug")
|> do_something_else()

Rule of thumb: if IO.inspect is the last expression in a function, be explicit about what you want to return.

Strategic Placement with the Pipe Operator

Because IO.inspect returns its argument, it fits perfectly in the middle of a pipeline:

users
|> IO.inspect(label: "Before filtering")
|> Enum.filter(& &1.active)
|> IO.inspect(label: "After filtering")
|> Enum.map(& &1.name)

This lets you peek into the data flow at exactly the point you want, without rewriting your code into intermediate variables.

A neat trick is to place multiple IO.inspect calls at different stages, each with a distinct label, so you can see how the data changes step by step.

Using Labels for Clarity in Elixir

Without labels, multiple inspection outputs can be hard to tell apart. The label option solves this:

IO.inspect(data, label: "After filtering")

Prints:

After filtering: [%{name: "Alice"}, %{name: "Bob"}]

When debugging a pipeline with several inspect points, labels make the output self-describing. This is especially useful when you're debugging multiple similar data structures in the same run.

Pretty Printing and Formatting Output

Sometimes, especially with large maps or deeply nested lists, the default single-line output is hard to read. That's where formatting options defined in Inspect.Opts come in.

Option	Description	Default
`:pretty`	If set to `true`, enables pretty printing.	`false`
`:limit`	Limits the number of items inspected for tuples, bitstrings, maps, lists and any other collection of items, with the exception of printable strings and printable charlists which use the `:printable_limit` option. If you don't want to limit the number of items to a particular number, use `:infinity`. It accepts a positive integer or `:infinity`.	`50`
`:printable_limit`	Limits the number of characters that are inspected on printable strings and printable charlists. You can use `String.printable?/1` and `List.ascii_printable?/1` to check if a given string or charlist is printable. If you don't want to limit the number of characters to a particular number, use `:infinity`. It accepts a positive integer or `:infinity`.	`4096`
`:width`	Number of characters per line used when pretty is `true` or when printing to IO devices. Set to `0` to force each item to be printed on its own line. If you don't want to limit the number of items to a particular number, use `:infinity`.	`80`
`:charlists`	When `:as_charlists`, all lists will be printed as charlists, non-printable elements will be escaped. When `:as_lists`, all lists will be printed as lists. When the default `:infer`, the list will be printed as a charlist if it is printable, otherwise as a list. See `List.ascii_printable?/1` to learn when a charlist is printable.	`:infer`

Example:

data = for i <- 1..100 do
  %{id: i, value: String.duplicate("x", i)}
end

IO.inspect(data, pretty: true, limit: 5)

Prints:

[
  %{id: 1, value: "x"},
  %{id: 2, value: "xx"},
  %{id: 3, value: "xxx"},
  %{id: 4, ...},
  %{...},
  ...
]

Here, we can see the first five entries, and the rest are summarized with ....

Coloring and Styling Output

Elixir's inspect options support syntax coloring: very handy when your terminal is full of logs.

Example:

IO.inspect(data,
  syntax_colors: [
    atom: :blue,
    string: :green,
    number: :red
  ],
  pretty: true
)

This makes atoms blue, strings green, and numbers red in your console output.

Colors can be any IO.ANSI.ansidata/0 as accepted by IO.ANSI.format/1.

If you want to use the default colors (like what is used in IEx), you can use IO.ANSI.syntax_colors/0:

IO.inspect(data,
  syntax_colors: IO.ANSI.syntax_colors(),
  pretty: true
)

Note: The colors are only visible if your terminal supports ANSI colors.

Conditional Inspection

Sometimes you only want to inspect if a certain condition is true: for example, when a debug flag is enabled or when a value crosses a threshold.

Here's an example with a debug flag:

def debug(term) do
  if Application.get_env(:my_app, :debug) do
    IO.inspect(term, label: "Debug")
  end
  term
end

You can use the function like this:

users
|> Enum.filter(& &1.active)
|> debug()
|> Enum.map(& &1.name)

You can toggle the output by setting config :my_app, :debug, true or false.

Capturing Inspect Output Instead of Printing

If you want to get the inspected form of a value without printing it, use inspect/2 (without the IO. prefix):

string_representation = inspect(data, pretty: true)

This is useful if you want to:

Write the debug output to a file
Send it to a logging service
Include it in an exception message

Example:

File.write!("debug.log", inspect(data, pretty: true))

You can also redirect IO.inspect output to another device:

IO.inspect(data, label: "Debug", device: :stderr)

Debugging Concurrency and Async Code

When you use IO.inspect in async code, the output may arrive out of order, making it hard to follow. Adding identifiers or timestamps can help.

Example:

tasks =
  for id <- 1..3 do
    Task.async(fn ->
      IO.inspect(self(), label: "PID #{id}")
      :timer.sleep(100)
      id * id
    end)
  end

Enum.map(tasks, &Task.await/1)

Prints:

PID 3: #PID<0.112.0>
PID 2: #PID<0.111.0>
PID 1: #PID<0.110.0>

Including the PID or a unique request ID in your label helps you trace which output belongs to which process.

Advanced Trick: Inspecting and Pattern Matching in One Go

You can combine pattern matching with inspection to see exactly what's being matched:

%{id: id} = IO.inspect(user, label: "User before insert")

This inspects the user variable before pattern matching extracts the id.

You can also place IO.inspect inside a guard to conditionally print:

case user do
  %{role: :admin} = u -> IO.inspect(u, label: "Admin user")
  _ -> :ok
end

In this last example, "Admin user" will only be printed if the user has the role :admin.

Using `dbg/2` for Richer Inspection

Since Elixir 1.14, we have dbg/2, a built-in debugging helper that works like IO.inspect but also shows the code expression that produced the value.

You can use it like this:

users |> Enum.filter(& &1.active) |> dbg()

Which prints:

users #=> [%{active: false, name: "Alice"}, %{active: true, name: "Bob"}]
|> Enum.filter(& &1.active) #=> [%{active: true, name: "Bob"}]

This makes it much easier to understand where in your code the inspected value is coming from, especially when you're inspecting multiple similar-looking values.

You can also pass options similar to IO.inspect:

dbg(users, label: "Active users", pretty: true)

Here are the key differences with IO.inspect:

Shows the code expression automatically.
Output format is slightly more verbose.
Same return behavior, returns the inspected value, so you can keep it in a pipeline.
Best for interactive debugging during development.

When `IO.inspect` is Not Enough

While IO.inspect is a fantastic quick-and-dirty tool, there are times when you need more powerful debugging:

IEx.pry: drops you into an interactive REPL inside the running process.
:observer.start/0: Erlang's GUI for monitoring processes, memory, and more.

The trick is to know when to reach for IO.inspect and when to switch to one of these other tools.

Setting Default Options for `IO.inspect` in IEx

When using IEx, you can configure the default options for IO.inspect using the IEx.configure/1 function:

IEx.configure(
  colors: [
    syntax_colors: [
      number: :light_yellow,
      atom: :light_cyan,
      string: :light_black,
      boolean: :red,
      nil: [:magenta, :bright]
    ],
    ls_directory: :cyan,
    ls_device: :yellow,
    doc_code: :green,
    doc_inline_code: :magenta,
    doc_headings: [:cyan, :underline],
    doc_title: [:cyan, :bright, :underline]
  ]
)

You can easily put this in your .iex.exs file so that it's applied automatically every time you open the shell.

Creating a Reusable Inspect Helper

Here's a neat way to wrap IO.inspect/2 into your own helper module with preferred defaults. This way, you can keep consistent inspection output without repeating options everywhere:

defmodule MyApp.Debug do
  @moduledoc """
  Convenience wrapper around `IO.inspect/2` with sensible defaults.
  """

  @default_opts [
    label: "DEBUG",
    pretty: true,
    limit: :infinity,
    width: 120,
    syntax_colors: IO.ANSI.syntax_colors()
  ]

  @doc """
  Inspects a value with default debug options.

  This is pipe-friendly: it returns the given value unchanged
  after inspecting it.

  ## Examples

      iex> [1, 2, 3]
      ...> |> Enum.map(&(&1 * 2))
      ...> |> MyApp.Debug.inspect()
      [2, 4, 6]
  """
  def inspect(term, opts \\ []) do
    term
    |> IO.inspect(Keyword.merge(@default_opts, opts))
  end
end

You can use it like this:

result =
  users
  |> Enum.filter(&(&1.active?))
  |> MyApp.Debug.inspect(label: "Active users")
  |> Enum.map(& &1.name)
  |> MyApp.Debug.inspect(label: "User names")

This way:

You get pretty printing (pretty: true) by default.
Lists and maps are not truncated (limit: :infinity).
A label is always shown (DEBUG unless overridden).
You can still override any option per call.
It works fine when it's in a pipeline.

Handy Visual Studio Code snippets

To make using IO.inspect faster and more consistent, you can configure editor snippets in Visual Studio Code so you don't have to type repetitive boilerplate each time.

In Visual Studio Code, open: Code -> Settings… -> Configure Snippets -> Elixir

Add the following snippet definitions:

{
  "inspect": {
    "prefix": "lin",
    "body": "IO.inspect($1, label: \"$1\", pretty: true)",
    "description": "IO.inspect with label."
  },
  "inspectSelectedText": {
    "prefix": "sin",
    "body": "IO.inspect($TM_SELECTED_TEXT, label: \"${TM_SELECTED_TEXT/(.*)/${1:/upcase}/}$0\", pretty: true)",
    "description": "IO.inspect with selected text as label."
  },
  "pipeInspect": {
    "prefix": "pin",
    "body": "|> IO.inspect(label: \"$1\", pretty: true)",
    "description": "IO.inspect with pipe."
  },
  "pipeInspectWithFileReference": {
    "prefix": "pinf",
    "body": "|> IO.inspect(label: \"$TM_FILEPATH:$TM_LINE_NUMBER$0\", pretty: true)",
    "description": "IO.inspect with pipe and file reference."
  },
  "inspectFromClipboard": {
    "prefix": "cin",
    "body": "IO.inspect($CLIPBOARD, label: \"${CLIPBOARD/(.*)/${1:/upcase}/}$0\", pretty: true)",
    "description": "IO.inspect clipboard content."
  }
}

With these in place, you'll have short prefixes to insert commonly used IO.inspect/2 patterns:

Prefix	Description
`lin`	`IO.inspect` with label
`sin`	`IO.inspect` using selected text as label
`pin`	`IO.inspect` in a pipeline
`pinf`	`IO.inspect` in a pipeline with file and line reference
`cin`	`IO.inspect` clipboard content

This way, you can quickly drop in debug output with consistent formatting, colors, and labels, without breaking your flow.

Suppose you are transforming a list of user maps and want to check intermediate results inside a pipeline:

users
|> Enum.filter(&(&1.active))
|> IO.inspect(label: "After filter", pretty: true)
|> Enum.map(& &1.email)

With the snippet defined above, you don't have to type all that. Just type pin, hit Tab, and you get:

|> IO.inspect(label: "", pretty: true)

You can immediately type your label (e.g., "After filter") and continue coding. This keeps your debugging consistent, colorful, and fast, without breaking your flow.

Best Practices

Now, let's finally look at a few best practices when using IO.inspect.

Use labels liberally: Unlabelled output is harder to parse.
Limit output: Use limit and pretty for large collections.
Avoid leaving it in production unless intentional.
Wrap it in helper functions when you want conditional control.
Tag concurrent output with PIDs, timestamps, or request IDs.
Credo can be used to detect unintentional calls to IO.inspect in a CI/CD pipeline.

Wrapping Up

IO.inspect may look like a humble debugging tool, but in Elixir, it's a powerful way to see what's going on without breaking your code's flow. By combining it with labels, formatting, conditional output, and process context, you can get precise insights into your program's behavior, all without leaving your editor.

Happy coding!

Structs and Embedded Schemas in Elixir: Beyond Maps

Roy Tomeij — 2025-09-09T00:00:00+00:00

If you work with Elixir, chances are you've used structs plenty of times and are likely aware of Ecto schema.

However, you may not have explored structs in depth or used Ecto schemas beyond the database context.

In this post, we'll explore the ins and outs of structs and Ecto schemas.

What is a Struct?

Visually, a struct is the closest thing Elixir has to a class. Functionally, it's a named map with some extra features. You'd typically use it in place of a map when your code might benefit from those extra features.

The defstruct macro in Kernel.Utils is fairly large, but mostly understandable.

Generally, a struct relies on some built-in Elixir features to enhance a map by:

Stopping you from calling defstruct in the same module twice
Setting up a layer of checks to ensure you only access and use specific keys in a struct
Setting up optional key enforcement

Some of these features use :ets, Erlang’s in-memory table system (which warrants its own article).

So eventually, you end up with an :ets-powered map that has a few extra hidden keys and special features. These features make structs a bit stricter, better defined, and, in some ways, more powerful than plain maps.

Let's see how structs work in practice.

What Happens Behind the Scenes with Structs in Elixir?

Let's start by defining a struct:

defmodule User do
  defstruct [:name, :email, :age]
end

With that small bit of code, we've declared a special kind of map. Where a regular map would look like this:

map_user = %{name: "John", email: "john@example.com", age: 30}

A struct is declared like this:

user = %User{name: "John", email: "john@example.com", age: 30}

Under the hood, the struct is still a map, but has an extra key:

iex> Map.keys(user)

[:name, :__struct__, :email, :age]

iex> user.__struct__

User

The __struct__ field is automatically added and helps Elixir distinguish between regular maps and structs.

Structs can have strict or required keys: let's look at those next.

Structs Have Strict Keys

Structs provide compile-time guarantees about which keys are valid.

This works:

%User{name: "John", email: "john@example.com"}

But this raises an error:

%User{name: "John", foo: "value"}

** (KeyError) key :foo not found
    expanding struct: User.__struct__/1

Accessing a value through a key actually works in the same way as with maps:

user.foo

# ** (KeyError) key :foo not found in: %User{name: "John", email: "john@example.com", age: 30}

map_user.foo

# ** (KeyError) key :foo not found in: %{name: "John", age: 30, email: "john@example.com"}

Accessing using [], however, is very different:

map_user[:foo]

# nil

user[:foo]

# ** (UndefinedFunctionError) function User.fetch/2 is undefined (User does not implement the Access behavior
# You can use the "struct.field" syntax to access struct fields. You can also use Access.key!/1 to access struct fields dynamically inside get_in/put_in/update_in). Make sure the module name is correct and has been specified in full (or that an alias has been defined)

So using Access behavior doesn’t add extra enforcement. Accessing an invalid key fails simply because structs don’t implement the Access protocol.

Matching now works like this:

%{foo: foo} = user

# ** (MatchError) no match of right hand side value:
#    %User{name: "John", email: "john@example.com", age: 30}

%{foo: foo} = map_user

# ** (MatchError) no match of right hand side value:
#    %{name: "John", age: 30, email: "john@example.com"}

So it works mostly in the same way, but there is a slight difference if you include the struct name in the match (which we will get to).

Structs Can Have Required Keys

You can also enforce required keys. If we define our struct like this:

defmodule User do
  @enforce_keys [:email]
  defstruct name: nil, email: nil, age: nil
end

This is possible:

user = %User{email: "john@example.com"}
user = %User{email: "john@example.com", age: 25}
user = %User{email: "john@example.com", name: "John"}
user = %User{email: "john@example.com", name: "John", age: 25}

But not giving an email when creating a struct raises an error:

user = %User{age: 30}

# ** (ArgumentError) the following keys must also be given when building struct User: [:email]
#     expanding struct: User.__struct__/1

It's a bit of extra strictness you can give your struct for various use cases.

Structs Can Have Initial Values

This doesn't have a wide application, but can be useful for a bunch of stuff (for example, when you're using structs to power configuration options):

defmodule User do
  defstruct [name: nil, email: nil, age: 20]
end

%User{}
# User{name: nil, email: nil, age: 20}

It also acts as the basis for Ecto schemas.

Structs Can Be Pattern-matched in Function Clauses

Structs enable a bit of extra pattern matching that goes beyond what maps offer. Here, we define a few basic function clauses for our user struct:

def process_user(%User{age: age}) when age >= 18 do
  "Adult user"
end

def process_user(%User{age: age}) when age < 18 do
  "Minor user"
end

def process_user(%User{name: name, email: email}) do
  "User #{name} with email #{email}"
end

The compiler can statically verify that you're matching on valid struct fields, catching typos and invalid keys at compile time.

You can't just call process_user/1 with any map that has :name, :email, or :age keys. It has to be the User struct. Not some random map, not some random other struct, specifically the user struct.

Really, the User part of %User{...} is a form of pattern matching. It means that it's a map, that has the __struct__ key, the value of which is User.

That means you can do more with it.

The following clause accepts a %User{} and a %PowerUser{}:

def process_user(%m{name: name} = user) when m in [User, PowerUser] do
  "User #{name} is a #{m}"
end

And the following accepts any struct, but not regular maps:

def process_user(%_{name: name} = user) do
  "User #{name} is a #{m}"
end

Note that matching in function clauses follows the same rules with structs as anywhere else. You can't specify keys that the struct doesn't define.

This wouldn't work, and would raise a CompileError:

def process_user(%User{foo: foo}), do: nil

Structs Can Have Dialyzer Types Declared

If your project uses Dialyzer, a common pattern (as suggested in the official documentation) is to declare a type t within the struct module.

In our example, we get User.t():

defmodule User do
  defstruct [:name, :email, :age]

  @type t :: %__MODULE__{
    name: String.t() | nil,
    email: String.t() | nil,
    age: non_neg_integer() | nil
  }
end

@spec create_user(String.t(), String.t(), non_neg_integer()) :: User.t()
def create_user(name, email, age) do
  %User{name: name, email: email, age: age}
end

Dialyzer can use these type specifications to catch type mismatches and provide better static analysis.

With the advent of actual types in elixir, this is slowly getting phased out. Once we have real type declarations, Dialyzer typespecs will probably get replaced. For now, though, it's a useful pattern to follow.

Structs Are Stricter, More Powerful Maps

For a bit of extra boilerplate, structs give you:

More power: Compile-time validation, enforced keys, and better pattern matching.
Extra checks: Static analysis of field names and types.
Clarity and self-documentation: Clear contracts about data shape.

Now let's move on to look at schemas.

What is an Embedded Schema?

There are two types of Ecto schemas:

Regular schemas are backed by a database table or view, and used to map database tables to Elixir code.
Embedded schemas are not directly associated to a database table and were originally intended to power jsonb columns in databases using embeds_one, embeds_many, etc.

Very obviously, both were originally intended to power Ecto and work with database tables.

With the split of ecto and ecto_sql, embedded schemas started to get used for much more.

Embedded Schemas Behind the Scenes

An embedded schema is essentially a struct with additional Ecto functionality. Here's how you declare one:

defmodule Address do
  use Ecto.Schema

  embedded_schema do
    field :street, :string
    field :city, :string
    field :postal_code, :string
    field :country, :string, default: "US"
  end
end

This creates a struct similar to what we saw before, but with Ecto's schema capabilities layered on top.

What actually happens is not a huge amount, but a bit more than what we get with just structs.

The line use Ecto.Schema first calls the __using__ macro, which registers a bunch of accumulating module attributes that the subsequent steps will use.

A module attribute is a named value you can use to store data within a module. Here's an example:

defmodule MyModule do
  @foo "bar"
  def get_foo, do: @foo

  @foo "baz"
  def get_foo_also, do: @foo
end

Here, get_foo() will return "bar", while get_foo_also() returns "baz". However, if you set the attribute to accumulate, using this line in the same module:

Module.register_attribute(__MODULE__, :foo, accumulate: true)

Then extra values set to the attribute will no longer replace the old value, but rather append to a list. So get_foo_also() in the same example will return ["bar", "baz"].

Eleven such attributes in total are registered, their intent being to hold the schema definition: fields, associations, primary keys, etc. This is then used by the embedded_schema macro.

The embedded_schema macro simply calls the schema macro with the source argument set to nil. So instead of:

embedded_schema do
  # ...
end

You could actually just do:

schema nil do
  # ...
end

Really, the macros do exactly the same thing, other than not have a source.

When defining a schema using either of these macros, the following happens:

A bit of code is injected into the module to import the various helpers we need, such as field, embeds_one, etc.
The block we pass into the macro contains calls to the above helpers. This code now runs and puts values into the accumulating attributes registered, storing all the schema information we specified for later use.
The module now contains all the schema information within its attributes. The Ecto.Schema.__schema__/1 function is called, and the module is passed in as an argument. This gives us a tuple containing all of our struct's fields and something called bags_of_clauses.
defstruct is called with the list of struct fields, defining a plain struct.
__changeset__ and __schema__ functions are defined on the module. The __changeset__ function is what allows our schema struct to be used with Ecto changesets, and the schema function provides introspection for us as well as for usage with Ecto.Query. The bags_of_clauses value defines a few of the clauses for the __schema__ function.

Our Address embedded schema effectively becomes this struct:

defstruct [street: nil, city: nil, postal_code: nil, country: "US"]

Note that @enforce_keys is not set, but you could still set it if you add it right before the call to embedded_schema.

So, effectively, embedded_schema creates a struct, but with extra features that make it work with Ecto.

Changesets and Validation

Embedded schemas shine when combined with changesets:

defmodule Address do
  use Ecto.Schema

  embedded_schema do
    field :street, :string
    field :city, :string
    field :postal_code, :string
    field :country, :string, default: "US"
  end

  def changeset(address, attrs) do
    address
    |> cast(attrs, [:street, :city, :postal_code, :country])
    |> validate_required([:street, :city, :postal_code])
    |> validate_format(:postal_code, ~r/^\d{5}(-\d{4})?$/)
  end
end

This provides type enforcement, format validation, and required field checks. Your changeset will check that all the required fields have a value, validate the format of the postcode, and will mark itself as invalid if anything is invalid.

This is clearly originally intended for databases, but because the embedded schema doesn't need a backing table, it can be used for much more.

API Input Prevalidation and the Command Pattern

Embedded schemas with changesets provide a cheap way to parse and sanitize API input. I usually call these special embedded schemas "commands", in that they command our business logic to do something.

defmodule CreateUserCommand do
  use Ecto.Schema

  alias Ecto.Changeset

  embedded_schema do
    field :name, :string
    field :email, :string
    field :age, :integer
  end

  def changeset(params) do
    %__MODULE__{}
    |> cast(params, [:name, :email, :age])
    |> validate_required([:name, :email])
    |> validate_format(:email, ~r/@/)
    |> validate_number(:age, greater_than: 0)
  end

  def validate(params) do
    case changeset(params) do
      %Changeset{valid?: true} = changeset ->
        {:ok, Changeset.apply_changes(changeset)}
      %Changeset{valid?: false} = changeset ->
        {:error, changeset}
    end
  end
end


defmodule UserController do
  def create(conn, %{"user" => user_params}) do
    with {:ok, command} <-  CreateUserCommand.validate(user_params) do
      UserService.create_user(command)
      send_resp(conn, 200)
      # the FallbackController handles the error response
    end
  end
end

This is something you would generally use for checks and parameter sanitization that don't need access to the database (which tends to be true for most of the checks you would want to do.)

For example, when creating a user, you might need to check that the email is unique, that any associated foreign keys exist in the database, or that a number is greater than another number in the database. Those all need a database check.

But there are things you can use an embedded schema for, to prevalidate the insert by just checking and parsing the params, such as:

The email being a string and of a valid format
The first name being provided
Age being a number between 18 and 30

Basically, anything that enforces basic constraints and rules you know the input should obey.

That way, you can do a fast and cheap validation, reject the request if it doesn't pass, and avoid wasting resources by doing all the necessary validations at once (both cheap and expensive ones). This enables a fail fast approach in your API.

The added bonus of this pattern is that if your changeset is valid, you can use Changeset.apply_changes/1 to get the CreateUserCommand struct with all of the data set, ready to be passed into the deeper business logic.

So instead of one big complex thing, you get two simpler things, and the more important, more expensive one can now be reused in other places (like, for example, in your LiveView admin dashboard).

Powering Forms in Live (and Dead) Views

LiveView already promotes using %Form{} structs to power forms. These structs are basically wrappers around changesets. Converting a changeset to a form is as simple as calling Phoenix.Component.to_form(changeset).

So understanding that the schemas powering those changesets don't need to be tied to a table unlocks a lot you could do.

Say you have a LiveView page with a list of users, and a form to add a new user. You can create a changeset for that form as early on as in your mount function.

def mount(_params, _session, socket) do
  changeset = CreateUserCommand.changeset(%{})
  {:ok, assign(socket, changeset: changeset)}
end

Then you render the form passing in to_form(@changeset), binding change and submit events.

def user_form(assigns) do
  ~H"""
  <.form
    for={to_form(@changeset)}
    :let={f}
    phx-change="validate"
    phx-submit="create"
  >
    <.input type="text" :field={f[:name]} />
    <.input type="email" :field={f[:email]} />
    <.input type="number" :field={f[:number]} />
  </.form>
  """
end

This isn't really new. We've used changesets to power forms since before LiveView, but the key point is that the schema powering the changeset doesn't need to be tied to a database.

def handle_event("validate", %{"user" => params}, socket) do
  changeset =
    params
    |> CreateUserCommand.changeset()
    |> Map.put(:action, :validated)  # This makes errors visible

  {:noreply, assign(socket, changeset: changeset)}
end

def handle_event("create", %{"user" => params}, socket) do
  case CreateUserCommand.validate(params) do
    {:ok, command} ->
      # this is live view. we expect it will work as the form is correctly setup and prevalidation worked
      # if it fails, let it crash, elixir style!
      {:ok, user} = UserService.create_user(command)
      {:noreply, put_flash(socket, "User created!")}
    {:error, changeset} ->
      {:noreply, assign(socket, changeset: Map.put(changeset, :action, :validated)}
  end
end

Instead, we've reused our CreateUserCommand, so that our API and LiveView share roughly two-thirds of the code.

If the command doesn't suit your form, you can always define a custom embedded schema module right in your LiveView, or even use schemaless changesets.

The same principle applies. The only difference with schemaless changesets is that we need to give them a name when converting them to a form using to_form(changeset, as: :my_name).

Important: Validation errors aren't visible until the changeset's :action field is something other than nil. When Repo.insert(changeset) or Repo.update(changeset) get called, the returning invalid changeset in {:error, changeset} has the action field set to insert or update, respectively.

This never happens with changesets based on embedded schemas, so you need to set the action manually using Map.put, for example.

A Word of Caution About Elixir Packages `typed_struct` and `domo`

The typed_struct package eliminates some boilerplate by automatically generating type specifications:

defmodule User do
  use TypedStruct

  typedstruct do
    field :name, String.t()
    field :email, String.t()
    field :age, non_neg_integer()
  end
end

This is effectively the same as declaring a User struct using defstruct, enforcing all keys and declaring a User.t() type.

However, this package hasn't been receiving updates recently, which is a concern for long-term maintenance. Plus, with the advent of Elixir types, its usefulness is likely to decrease.

The domo package builds on top of typed structs by generating utility functions, such as new and new!. But, in my experience, it also adds a significant compilation time overhead.

At V7, we've been using these extensively, and they've proven useful.

They've also proven to increase our compilation time significantly, and domo seems to regularly cause compilation deadlocks, so we're seriously considering removing them.

Let's finally take a quick look at future Elixir types before wrapping up.

Future Elixir Types

With the growth of the Elixir type system, structs are becoming safer and safer. The compiler is inferring more and catching more bugs.

We're likely approaching a point where it becomes questionable whether we should use Dialyzer, for example.

The way I see it, Dialyzer is as useful as ever, and whatever type of improvements we get in the background are free and do not conflict with it at all. If anything, they might reveal where we can improve our type specs.

Once Elixir type specifications become available, we will probably be replacing type specs with those, and hopefully, that will be an easy migration.

In the meantime, we should lean into things that already benefit from the type system and offer the best of both worlds.

Wrapping Up

In this post, we ran through some use cases for Elixir structs and schemas.

Hopefully, you now have a deeper understanding of structs, even if you were already comfortable using them.

As for embedded schemas, maybe you've found a new use case for them.

Happy coding!

Tracking Errors in Absinthe for Elixir with AppSignal

Roy Tomeij — 2025-08-19T00:00:00+00:00

GraphQL provides a powerful approach to building APIs, and Absinthe is the leading GraphQL implementation for Elixir applications. While GraphQL offers many benefits, it can introduce a set of errors and performance bottlenecks that might be challenging to track and debug.

In this article, you’ll learn how to use AppSignal to monitor, debug, and resolve errors in your Absinthe-based GraphQL API.

To follow along with the tutorial, you'll need a few things in place.

Prerequisites

An AppSignal account — you can sign up for a free trial.
Some experience working with Elixir and the Phoenix framework.
A local installation of Elixir, the Phoenix framework, and PostgreSQL. Follow this guide to get your local environment set up.
An Elixir app with Absinthe integrated. If you don't have one ready, fork the example app we'll be using for this tutorial.

Introducing Our Example Phoenix App

Our example application is a Phoenix app that models a simple sports league management system with leagues, seasons, teams, and players. The API is accessible via a GraphQL endpoint at /wendigo/api/graphql, and a GraphiQL interface at /wendigo/api/graphiql (a nifty web interface for testing the API in development).

Going through a step-by-step project build would make this tutorial too long. So, from here onwards, we'll be referencing our pre-built example app, which already has Absinthe integrated. For more information on integrating Absinthe into a new Phoenix application, see the documentation.

Next, let's add the AppSignal mix package.

Adding the AppSignal Package

Open up the mix.exs file and add the main AppSignal package, as well as the AppSignal Phoenix package (required for Phoenix apps):

# mix.exs

...
defp deps do
  [
    ...
    {:appsignal, "~> 2.8"},
    {:appsignal_phoenix, "~> 2.0"}
    ...
  ]
end
...

Now run mix deps.get to fetch the newly added dependencies.

Installing and Configuring AppSignal

Run mix appsignal.install YOUR_PUSH_API_KEY to install AppSignal, then go through the configuration options that will be presented to you:

Name your app: Give your app a suitable name.
The configuration method to be used: Here, you have a choice between using an environment variable or using a config file. I recommend using the configuration file, as it provides more flexibility later on.

Once you're done with the configuration, you'll see an output similar to the one below, which means you've successfully configured AppSignal:

Welcome to AppSignal!

This installer will guide you through setting up AppSignal in your application.
We will perform some checks on your system and ask how you like AppSignal to be
configured.

================================================================================


Validating Push API key: Valid! 🎉
What is your application's name? [wendigo]: wendigo_app

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

What is your preferred configuration method? [1]: 1
Writing config file config/appsignal.exs: Success!
Linking config to config/config.exs: Success!
Activating dev environment: Success!
Activating prod environment: Success!

AppSignal detected a Phoenix app
...

Once that's done, the app should start sending data over to the AppSignal dashboard:

Before moving into the actual implementation of error tracking in Absinthe, it's very important that we learn what GraphQL is all about, and especially how it works. This will help us understand where errors could come from and give us a basis for setting up effective error tracking.

What Is GraphQL?

Nowadays, building applications often involves providing API access points. REST APIs are often used as the solution for serving data to clients, but creating a REST API that meets all user requirements can be challenging, especially as applications grow in complexity.

And that's where GraphQL comes in. Unlike REST, GraphQL empowers clients to request the exact data they need.

We won't go into the details of how GraphQL actually works; instead, we'll focus on the GraphQL schema since it's the interface concerned with processing GraphQL requests. Later in this article, we'll learn how errors occur and how to resolve them using AppSignal.

The GraphQL Schema

The GraphQL schema is a blueprint of the entire API. It helps us define the domain model and how the server will respond to requests for data.

Below is summarized code showing the example application schema:

#lib/wendigo_web/graphql/schema.ex

defmodule WendigoWeb.GraphQL.Schema do
  use Absinthe.Schema

  alias Wendigo.Context.{Leagues, Players, Seasons, Teams}
  alias WendigoWeb.Resolvers.{LeagueResolver, PlayerResolver, SeasonResolver, TeamResolver}

  import_types(Absinthe.Type.Custom)
  import_types(WendigoWeb.GraphQL.Types)

  query do
    @desc "Get a league"
    field :league, :league do
      arg(:id, non_null(:id))
      resolve(&LeagueResolver.get_league/3)
    end

    @desc "List leagues"
    field :leagues, list_of(:league) do
      arg(:first, :integer)
      arg(:offset, :integer)
      resolve(&LeagueResolver.list_leagues/3)
    end

    ...
  end

  mutation do
    @desc "Create a season"
    field :create_season, :season do
      arg(:name, non_null(:string))
      arg(:start_date, non_null(:date))
      arg(:end_date, non_null(:date))
      resolve(&SeasonResolver.create_season/3)
    end
    ...
  end
end

The schema is made up of several key components: types, queries, and mutations.

Types

Types describe the shape of the data objects. Types can be used directly in the schema or imported from elsewhere to keep the schema clean and organized.

The code below shows the league object type:

# lib/wendigo/schema/league.ex

defmodule Wendigo.Schema.League do
  @moduledoc """
  Schema for the `leagues` table.
  """
  use Ecto.Schema
  import Ecto.Changeset

  @type t() :: %__MODULE__{}

  @primary_key {:id, :string, autogenerate: {Ecto.Nanoid, :autogenerate, []}}
  schema "leagues" do
    field :name, :string
    field :level, :string
    timestamps()
  end

  @doc "Creates a changeset for a league"
  def changeset(struct, params) do
    struct
    |> cast(params, [:name, :level])
    |> validate_required([:name])
    |> validate_length(:name, max: 100)
    |> validate_length(:level, max: 100)
  end

  def changeset(params),
    do: changeset(%__MODULE__{}, params)
end

Queries

These are read-only operations that fetch data.

...
query do
  @desc "Get a league"
  field :league, :league do
    arg(:id, non_null(:id))
    resolve(&LeagueResolver.get_league/3)
  end
end
...

Mutations

Mutations are operations that modify data:

...

mutation do
  @desc "Create a season"
  field :create_season, :season do
    arg(:name, non_null(:string))
    arg(:start_date, non_null(:date))
    arg(:end_date, non_null(:date))
    resolve(&SeasonResolver.create_season/3)
  end
  ...
end
...

With the information we have so far, let's now explore some of the errors you might encounter when working with Absinthe and how you can use AppSignal to track them.

Absinthe Errors

There are many error types that can affect a GraphQL API, including:

Schema validation errors - These happen when the query doesn't conform to the rules defined in the schema (for example, requesting a field that doesn't exist in the schema):

{
  user(id: "123") {
    nonexistentField
  }
}

Query syntax errors happen when a client sends a malformed query, (for example, an incorrect query syntax, as shown below):

{
  user(id: "123" { # Missing closing parenthesis
    name
  }
}

Resolver errors can occur when a resolver fails to fetch data for one reason or another. (For example, failing to find a record or mishandling nil values):

resolve fn %{id: id}, _ ->
  case Repo.get(User, id) do
    nil -> {:error, "User not found"}
    user -> {:ok, user}
  end
end

Type mismatch errors occur when the data returned by a resolver fails to match what is defined by the schema. A good example is returning a string instead of an integer:

field :age, :integer do
  resolve fn _, _ -> {:ok, "twenty three"} end
end

Complexity limit errors: Absinthe allows you to set up limits on the number of levels when deep nesting a query. As such, should a request exceed the set limits, a complexity limit error occurs.

{
  users {
    orders {
      items {
        product {
          supplier {
            contact {
              phone
            }
          }
        }
      }
    }
  }
}

These aren't the only errors you can expect; there are others like authentication (and authorization) errors, and argument validation errors. You can get more information about them from the GraphQL error documentation.

Now that you know the types of errors to expect, let's see how we can use AppSignal to handle a few of them.

Tracking Absinthe Errors Using AppSignal

By default, AppSignal will automatically instrument Absinthe requests, as you can see below:

However, for Absinthe-specific errors, you'll need an extra setup to effectively surface these errors to the AppSignal dashboard.

How Absinthe Treats Errors

Absinthe doesn't crash when resolvers return an error; instead, it includes these errors in the GraphQL response with a 200 HTTP status code.

You can see this in the example shown below, where I made a query to create a league and intentionally failed to include the required name field:

As you can see, returning errors as tuples allows the developer to handle expected failures whichever way they wish, without crashing the request flow.

For an exception to be raised in Absinthe, the following conditions need to be met:

Having actual bugs in the code. For example, undefined function modules and so forth.
Database or Ecto errors that are not handled at the context level.
Absinthe middleware errors.

These are not necessarily the only conditions, but they represent a big percentage of what would cause Absinthe to raise exceptions.

With that in mind, let's use an Absinthe middleware to set up a custom instrumentation and catch a resolver error that would have otherwise gone under the radar.

Custom Instrumenting Absinthe Errors

Let's assume we want to surface a resolver error when a user tries to fetch a team without a valid ID.

The relevant get_team/3 resolver function is shown below:

defmodule WendigoWeb.Resolvers.TeamResolver do
  alias Wendigo.Context.Teams
  alias WendigoWeb.Error

  def get_team(_parent, %{id: id}, _resolution) do
    case Teams.get(id) do
      nil ->
        {:error, "Team not found: #{id}"}
      team ->
        {:ok, team}
    end
  end

  ...
end

But like I mentioned earlier, this wouldn't show up as an error on AppSignal since Absinthe would still return a 200 status code. To make sure the error is surfaced as we want, we need to set up an Absinthe middleware.

Absinthe Middlewares

In Absinthe, a middleware functions as a layer that processes GraphQL requests as they flow through an application, allowing you to perform tasks like error handling, logging, authorization checks, or data transformation before or after resolvers execute.

For the purpose of error tracking, middlewares are particularly valuable because they can intercept and process any errors that occur during field resolution, whether they're from failed database queries, validation issues, or other runtime problems.

With that in mind, let's set up a middleware called ErrorReporter to handle resolution errors:

# lib/wendigo_web/middleware/error_reporter.ex

defmodule WendigoWeb.Middleware.ErrorReporter do
  @behaviour Absinthe.Middleware

  def call(resolution, _config) do
    case resolution.errors do
      [] -> resolution
      errors ->
        if span = Appsignal.Tracer.current_span() do
          formatted_errors = Enum.map(errors, &format_error/1)
          error_message = Enum.join(formatted_errors, ", ")

          error = %RuntimeError{
            message: error_message
          }

          Appsignal.Span.add_error(
            span,
            error,
            %{
              graphql_error: true,
              validation_error: true
            }
          )

          Appsignal.Span.set_sample_data(span, "graphql", %{
            query: resolution.definition.name || "anonymous",
            path: Absinthe.Resolution.path(resolution)
          })
        end

        resolution
    end
  end

  defp format_error(%{message: message}), do: message
  defp format_error(error), do: inspect(error)
end

Then, edit the schema to include this middleware:

# lib/wendigo_web/graphql/schema.ex

defmodule WendigoWeb.GraphQL.Schema do
  ...

  # Middleware
  def middleware(middleware, _field, _object) do
    middleware ++ [WendigoWeb.Middleware.ErrorReporter]
  end
end

Now let's see how this works:

The ErrorReporter middleware is added to all schema operations through the middleware/3 function shown in the code snippet above.
When get_team/3 is called, the middleware receives a resolution object for each GraphQL field. If there are no errors, the resolution proceeds.
If there are errors, the middleware formats each error as a string; multiple errors are joined together using commas and sent to AppSignal using the current span.

Having mentioned AppSignal's "span", here's what it is and how it works:

Appsignal.Span represents a single unit of work in a tracing call. The add_error() function attaches error information to the span as well as any metadata tags that might be available. In the same token, the set_sample_data() function is used to add context about a GraphQL request to enrich the data that will appear on AppSignal when the span data is sent.
Appsignal.Tracer is used for tracking the request flow in the app. The current_span() function returns the active trace span if it's available.

Now, whenever a resolver is executed, the middleware will run with the ability to catch and report errors from field resolvers, input validation, schema validations, runtime errors, and so forth.

A good example of middleware in action is shown in the screenshots below, which highlight a resolution error that has been reported in the AppSignal dashboard:

There's more detail about the error here:

Wrapping Up

In this article, we've explored how to effectively track and monitor errors in an Absinthe GraphQL API using AppSignal. We've covered the various types of GraphQL errors you might encounter, from schema validation to resolver errors, and demonstrated how to set up custom error tracking using Absinthe middleware.

Happy coding!

Understanding Stack Traces in Elixir

Roy Tomeij — 2025-08-05T00:00:00+00:00

If you have spent any time working with Elixir in a production setting — or even just building a side project — you've likely encountered a stack trace. Stack traces can appear intimidating at the outset, especially since Elixir's error messages are often more detailed than those in other languages. However, they are a powerful tool for understanding what went wrong in your code.

Early in my Elixir journey, I found stack traces to be a bit overwhelming. They seemed to contain a lot of information that I didn't know how to interpret. While we have the luxury of LLMs like ChatGPT to help us understand these traces today, it's still essential to know how to read and interpret them independently.

Effective debugging is crucial when developing applications for commercial or production use. Stack traces are a key tool in your debugging arsenal, helping you quickly identify the source of an error and trace it back to its origin.

As with many other aspects of the Elixir ecosystem, stack traces are shaped by the underlying principles of simplicity, explicitness, and fault tolerance.

In this article, we'll explore the fundamentals of stack traces in Elixir, how to read and interpret them, and some best practices for debugging and error handling.

First, let's define a stack trace.

What Is a Stack Trace?

I like to think of stack traces as a story that tells you how your program reached a specific breaking point. They're a snapshot of all the function calls that led up to an error, starting from the top-level function that was called.

At a high level, stack traces reflect a program's call history — a breadcrumb trail that shows how you got to where you are. Each line in a stack trace represents a function call, starting from the most recent and going back to the initial call that triggered the error.

However, there is a catch: Elixir runs on the BEAM VM, a platform optimized for concurrency and fault tolerance. An Elixir application is actually composed of thousands of lightweight processes that run concurrently, each on its own isolated stack. So, when an error occurs, the stack trace you see is only for the process that crashed, not the entire application.

This is a crucial concept to keep in mind. A stack trace tells the local story of a single process, not the global story of an entire application. Broader system behaviour emerges from how supervisors, links, and other OTP mechanisms interact with the crashed process.

Stack Traces in the BEAM (Erlang VM)

The BEAM's approach to error handling is simple but powerful: errors are expected, processes are allowed to fail, and systems should recover gracefully. Rather than burying exceptions or attempting to rescue every failure, Elixir (and Erlang before it) encourages developers to "let it crash" and rely on supervision trees to restart failed processes.

When something goes wrong in an Elixir application, one of three primary errors can occur: raise/1, exit/1, or throw/1. Each of these functions generates a stack trace that tells the story of how the error propagated through the system.

raise/1: This function is used to raise an exception, which can be caught by a try/rescue block. When an exception is raised, the stack trace will show the function call that raised the exception, along with the call history that led up to it.
exit/1: This function is used to terminate a process. When a process exits, it can send an exit signal to other processes, which may also generate a stack trace if those processes are linked to the exiting process.
throw/1: This function is used for non-local returns, allowing you to exit from a function and return to a different point in the call stack. It can also generate a stack trace, but its usage is less common than raise/1 and exit/1.

The BEAM VM is designed to handle errors gracefully, and stack traces are a key tool for understanding what went wrong and how to recover from it.

Key takeaway: Errors do not bubble up globally; they are local to processes. Supervisors and linked processes may receive notifications, but stack traces themselves belong to the failing process.

Differences Between Elixir and Erlang Stack Traces

Although Elixir runs on the BEAM VM, just like Erlang, both languages have very different approaches to presentation, semantics, and abstractions around stack traces. This is mostly due to Elixir's focus on developer experience and its goal of being a more approachable language for newcomers.

Presentation: Elixir stack traces are designed to be more human-readable than Erlang's. They include additional context, such as module and function names, and they format the output in a way that is easier to understand at a glance.
Semantics: Elixir's stack traces are more focused on the functional programming paradigm, while Erlang's stack traces are more aligned with the actor model. This means that Elixir's stack traces often include more information about function calls and their arguments, while Erlang's stack traces may focus more on the process and message-passing aspects of the system.
Abstractions: Elixir provides higher-level abstractions for error handling, such as try/rescue and try/catch, which are not present in Erlang. This means that Elixir's stack traces may include additional information about the context in which the error occurred, such as the specific clause that was matched or the arguments that were passed to the function.

Reading and Interpreting an Elixir Stack Trace

So far, we have covered the general concepts around stack traces and, specifically, how they are generated in the BEAM VM. But all that theory is useless if you don't know how to read a stack trace when you see one.

Let's take a look at a simple example of a stack trace generated by an Elixir application:

** (ArithmeticError) bad argument in arithmetic expression
 :erlang.+("foo", 1)
 my_app/lib/my_module.ex:10: MyModule.add/2
 (my_app 0.1.0) my_app.ex:5: MyApp.run/0

When an error occurs, Elixir prints out a stack trace that is comprised of the following elements:

Error Type: The type of error that occurred, in this case ArithmeticError.
Error Message: A description of the error, in this case bad argument in arithmetic expression.
Function Calls: A list of function calls that led to the error, starting from the most recent call and going back to the initial call that triggered the error.
Module and Function: The module and function where the error occurred, in this case MyModule.add/2.
Line Number: The line number in the source code where the error occurred, in this case my_app/lib/my_module.ex:10.

So, in this example, the error occurred in the add/2 function of the MyModule module, specifically on line 10 of the my_module.ex file. The error was caused by an attempt to add a string ("foo") to a number (1), which is not a valid operation in Elixir.

Let's take a look at another example, this time a GenServer crash:

** (FunctionClauseError) no function clause matching in MyApp.Worker.handle_call/3
 my_app/lib/my_app/worker.ex:15: MyApp.Worker.handle_call({:unknown, 123}, {#PID<0.142.0>, #Reference<0.0.4.6>}, %{})
 (stdlib 4.0) gen_server.erl:689: :gen_server.try_handle_call/4
 (stdlib 4.0) gen_server.erl:721: :gen_server.handle_msg/6

Note: Take a pause here to look at the stack trace and try to understand what went wrong on your own before reading the explanation below.

In this case, the error is a FunctionClauseError, which means that the function handle_call/3 in the MyApp.Worker module was called with arguments that did not match any of its defined clauses. The stack trace shows that the error occurred on line 15 of the worker.ex file, and it provides the specific arguments that were passed to the function: {:unknown, 123}.

The stack trace also shows the call history that led to the error, including the try_handle_call/4 and handle_msg/6 functions from the gen_server module. This information can help you understand how the error propagated through the system and where to look for potential fixes.

A few tips that can help you read and interpret stack traces more effectively:

Start from the top of the stack trace and work your way down. The first line usually contains the most relevant information about the error.
Stack traces often include both Elixir and Erlang layers. Focus on your own modules first, and then look at the Erlang functions if needed.
When chasing down an error, start from the topmost Elixir module in your codebase and work your way down.

Debugging Tools and Techniques in Elixir

So you have read the stack trace and you know where the error occurred, but how do you go about fixing it?

The following sections will cover some of the most common debugging tools and techniques available in Elixir.

Logging and Inspection

Sometimes you don't need an interactive debugger to figure out what is going on in your code. A simple IO.inspect/2 or Logger.debug/1 can go a long way in helping you understand the state of your application.

Elixir gives us a few options, but the most common ones are:

Use IO.inspect/2 for quick, inline data inspection:

IO.inspect(data, label: "Incoming data")

Logger.debug/1 or Logger.info/1 for production-aware logging:

require Logger
Logger.error("Unexpected state in handle_call: #{inspect(state)}")

Interactive Debugging

Sometimes you need to step through your code line by line to understand what is going on. Elixir provides a couple of tools for interactive debugging: dbg/2 and IEx.pry.

Use iex -S mix for live exploration. Reproduce bugs interactively.

Using `dbg/2`

The built-in dbg/2 macro (introduced in Elixir 1.14) allows inline debugging by printing both the value of an expression and the expression itself. This is particularly useful for quickly inspecting values during runtime without modifying too much of your code.

Example:

defmodule Example do
 def calculate(a, b) do
 dbg(a + b)
 end
end

Example.calculate(2, 3)
# Output:
# dbg(2 + 3) #=> 5

In this example, dbg/2 prints the expression 2 + 3 and its result 5. This makes it easier to understand what is happening in your code.

Using `IEx.pry`

For more advanced debugging, you can use IEx.pry to pause the execution of your code and interact with it in a live IEx session. This is similar to setting a breakpoint in traditional debuggers.

To use IEx.pry, you need to include the require IEx statement and call IEx.pry/0 where you want to pause execution.

Example:

defmodule Example do
 def calculate(a, b) do
 require IEx
 IEx.pry()
 a + b
 end
end

Example.calculate(2, 3)

When the code reaches IEx.pry/0, it will pause and open an interactive shell where you can inspect variables, evaluate expressions, and step through the code.

Note: To use IEx.pry, you must run your application with iex -S mix and ensure the :debugger application starts. If the debugger is not running, you will see a message like:

** (RuntimeError) cannot pry: the shell is not running or the debugger is not enabled

Raising, Catching, and Handling Exceptions

As mentioned, Elixir, like Erlang, embraces the "let it crash" philosophy.

Instead of trying to handle every possible error within a process, Elixir encourages developers to allow processes to fail and rely on supervision trees to restart them in a clean state. This approach simplifies error handling and ensures that the system remains robust and fault-tolerant.

However, there are scenarios where raising and catching errors is necessary. For example:

Input Validation: When user input or external data is invalid, raising an exception can provide clear feedback about what went wrong.
Boundary Layers: At the boundaries of your system (e.g., APIs, file I/O, or database interactions), catching errors allows you to handle failures gracefully and provide meaningful error messages to users or clients.
Critical Cleanup: In cases where resources (e.g., files, sockets, or database connections) need to be cleaned up, using try/after ensures that cleanup happens even if an error occurs.

Why Raise and Catch Errors?

While the "let it crash" philosophy works well for many scenarios, there are times when you need more control over error handling, including:

Graceful Degradation: In user-facing applications, you can catch errors to provide a fallback or a friendly error message instead of crashing the process.
Error Context: Raising custom exceptions with meaningful messages can make debugging easier by providing more context about what went wrong.
Controlled Recovery: In some cases, you may want to catch an error, log it, and retry the operation or take an alternative action.

For example:

def fetch_user_data(user_id) do
 try do
 # Simulate a risky operation
 raise "User not found" if user_id == nil
 {:ok, %{id: user_id, name: "John Doe"}}
 rescue
 e in RuntimeError -> {:error, e.message}
 end
end

In the example above, the function raises an exception if the user_id is nil. However, it also catches the exception and returns an error tuple, allowing the caller to handle the failure gracefully.

By combining the "let it crash" philosophy with deliberate error handling where appropriate, you can build resilient and user-friendly systems.

Raising Exceptions

Use raise/1 or raise/2 to deliberately trigger errors:

raise "Something went wrong!"
raise ArgumentError, "invalid argument passed"

Define custom exceptions for clarity:

defmodule MyApp.MyError do
 defexception message: "default message"
end

And raise it like this:

raise MyApp.MyError, message: "Custom error"

Catching Exceptions

You can catch errors using try/rescue:

try do
 risky_call()
rescue
 e in RuntimeError -> IO.puts("Caught: #{e.message}")
end

For throw/1, use try/catch:

try do
 throw(:error)
catch
 :throw, :error -> "Caught throw"
end

Use try/after to ensure cleanup happens:

try do
 File.read!("some_file.txt")
after
 IO.puts("Finished file operation")
end

Best Practice: Don't rescue all exceptions blindly. It breaks the "let it crash" principle and can hide bugs.

Rescuing exceptions truncates the stack trace to the rescue point. If you catch too early, you lose important debugging context.

Next Steps

Create a sandbox project and experiment with different error scenarios. Here are a few ideas that you can try:

Create a simple GenServer and trigger a FunctionClauseError by sending an unexpected message.
Raise an exception in a try block and see how the stack trace changes when you rescue it.
Use dbg/2 to inspect the call stack and see how it changes as you step through your code.
Protocols and behaviours are a great way to learn about stack traces. Create a protocol and implement it in different modules, then call the protocol function with different arguments to see how the stack trace changes.
Use Logger to log different levels of messages and see how they appear in the console.

In summary, treat errors as opportunities to learn and improve your code. Stack traces are a powerful tool for understanding what went wrong and how to fix it. By mastering stack traces and debugging techniques, you'll be better equipped to build robust and reliable Elixir applications.

Wrapping Up

In this article, we covered a lot of ground on stack traces and debugging in Elixir. Here are the key takeaways:

Stack traces in Elixir tell the story of a single process, not the entire application.
Elixir's "let it crash" philosophy means errors are expected and systems should recover gracefully.
To read stack traces effectively, start from the top and focus on your application's modules.
Debugging tools like IO.inspect/2, Logger, and interactive debugging can help you understand errors.
Exception handling should be purposeful and limited, preserving the valuable context that stack traces provide.
Well-designed supervision trees and meaningful error messages make production systems more resilient.

As with anything in programming, debugging is a skill that improves with practice. The more you work with Elixir and its stack traces, the more comfortable you'll become in reading and interpreting them.

An Introduction to Oban for Elixir Monitoring Using AppSignal

Roy Tomeij — 2025-07-22T00:00:00+00:00

Background task processing is something that many developers may encounter when building Elixir applications. This might include sending emails asynchronously, posting and fetching data from an API, and more.

Oban, a powerful and persistent job processing library, offers a reliable way to handle background tasks, scheduled operations, and more. However, like any complex system, Oban requires careful monitoring to ensure its smooth operation, identify bottlenecks, and prevent unexpected failures.

In this article, we'll learn about the fundamentals of Oban, its potential pitfalls, and how to leverage AppSignal's monitoring to gain valuable insights into your Oban setup.

Prerequisites

Elixir and the Phoenix framework installed on your local computer.
An AppSignal account. If you don't have one, sign up for a free trial.
An Elixir/Phoenix app to play around with. If you don't have one, you can fork this one that we'll use throughout this tutorial.

Introducing Oban

Oban is a robust background job processing library for Elixir that uses PostgreSQL, MySQL, or SQLite 3 for storing job queues. Unlike other job processing solutions, Oban doesn't require Redis or other external dependencies beyond your existing database.

Key Features of Oban

Persistence: Jobs are stored in the database, ensuring they survive application restarts.
Concurrency Control: Jobs can be organized into queues with configurable concurrency limits.
Scheduled Jobs: It offers support for one-off and recurring jobs with precision scheduling.
Job Prioritization: Within each queue, jobs can be assigned different priority levels.
Error Handling: Built-in retry mechanisms.
Unique Jobs: Oban provides mechanisms to prevent job duplication with uniqueness constraints.

In terms of structure, Oban consists of workers, which are the modules that define job processing logic, queues, and supervisors, used to manage job execution. This architecture makes it very good for reliable background processing in Elixir apps.

Introducing Our Example App and Setting up Oban

We'll use a simple email subscription app for this tutorial. A user will be able to submit their email address, and the app will then send a scheduled email using background job queues powered by Oban.

This app should provide enough context for you to learn about Oban and how to monitor it using AppSignal.

Adding Oban to an Elixir Project

Oban's installation is very straightforward. Begin by adding the following packages to your app's mix.exs file:

# mix.exs
...

{:oban, "~> 2.19"},
{:igniter, "~> 0.5", only: [:dev]},
...

Next, fetch the newly added dependencies with mix deps.get.

Installing Oban Using Igniter

Previously, after adding Oban and running the mix oban.install command, we had to modify the generated migration file manually.

But now, with the addition of the Igniter library, after running mix oban.install, the manual modification of the generated migration file is not needed.

Now that you know what Igniter is for, go ahead and install Oban with:

mix oban.install

Then run the generated migrations with mix ecto.migrate.

Adding AppSignal to the Project

Open up the mix.exs file, then edit it as shown below:

# mix.exs
...
defp deps do
    [
        ...
        {:appsignal, "~> 2.8"},
        {:appsignal_phoenix, "~> 2.0"}
    ]
end
...

Note: The AppSignal instrumentation for Phoenix apps is available through a separate appsignal_phoenix package, which depends on the core Elixir appsignal package.

Then run mix deps.get to fetch the package, followed by the command mix appsignal.install <YOUR APPSIGNAL API KEY> to install AppSignal.

You can choose how you want AppSignal configured for the project from the following options:

Customizing the name of your application.
Choosing the configuration method to be used (two options are available: via an environment variable or through the use of a configuration file).

Once you've made your choices, you should get an output similar to the one shown below if the process runs without issues.

Validating Push API key: Valid! 🎉
What is your application's name? [email_subscription_app]: email_subscription_app
There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)
What is your preferred configuration method? [1]: 1
Writing config file config/appsignal.exs: Success!
Linking config to config/config.exs: Success!
Activating dev environment: Success!
Activating prod environment: Success!
...
AppSignal installed! 🎉

Now you should have AppSignal installed, and your app will start sending data to your dashboard:

So far, we have a Phoenix application integrated with Oban and AppSignal. We can now turn our attention to the question of how to monitor Oban.

But before we do that, let's learn about some of Oban's challenges and potential points of failure for insights into where to focus our monitoring efforts.

Common Oban Challenges and Potential Points of Failure

Like any background job framework, Oban can encounter various challenges that affect performance and reliability. Understanding these potential issues is crucial for effective monitoring and maintenance. Let's explore some of these:

Queue overloading - When jobs are enqueued faster than they can be processed, queues can build up rapidly. This backlog can lead to increased memory usage, slower overall system performance, and delayed job execution. It is particularly problematic for time-sensitive operations.
Failed jobs and error handling - Jobs can fail for various reasons, including database timeouts, API rate limits, and so forth. Without proper error handling and monitoring, failed jobs might slip through the cracks, or worse, create a sequence of other failures.
Database contention - Since Oban uses your database for job storage, high job throughput can lead to database contention. This manifests as lock timeouts and increased query times and can impact your application's core functionality.
Worker crashes - When workers crash unexpectedly, jobs might be abandoned or stuck in a "running" state, and without proper supervision, these orphaned jobs can clog your system.

While these examples represent common challenges with Oban, they're just a few of the potential issues you might encounter.

With this knowledge of Oban's architecture and potential pitfalls, we're now ready to explore how AppSignal can help us gain visibility into our job processing system and detect problems before they impact users.

Monitoring Oban Using AppSignal

Once you've added the AppSignal package, it will automatically instrument Oban job workers and queue metrics out of the box.

Tip: AppSignal also adds an Oban namespace, which you can use to filter for Oban-specific metrics.

A few Oban metrics are automatically made available. Let's start by looking at job execution time.

Job Execution Time Metrics

This metric captures the duration spent by a worker to perform a particular job. For example, in the email subscription app, when a user submits an email, the WelcomeEmailWorker is used to send a welcome email.

The code below shows the Oban worker responsible for processing the welcome email:

# lib/email_subscription_app/workers/welcome_email_worker.ex

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()
    |> case do
      {:ok, _} -> :ok
      {:error, reason} ->
        raise "Mailer delivery error: #{inspect(reason)}"
    end
  end
end

And this is automatically captured on the AppSignal dashboard, as you can see below:

Metrics for Slow Jobs

AppSignal will also automatically capture slow jobs. The dashboard below shows you what this looks like:

AppSignal also automatically instruments Oban errors. Let's see how that looks next.

Automatic Metrics for Oban Errors

AppSignal provides excellent error instrumentation for Oban, as you can see from the screenshots below:

More details are made available when you click into a particular error:

While it's great having automatic instrumentation for some of the more common metrics, there will be times when you need to go beyond these.

Let's say you wanted to know the total number of welcome emails sent within a certain period of time or the number of failed ecommerce orders; such metrics might not be automatically captured. For custom metrics like these, it's necessary to implement custom instrumentation for Oban.

Custom Metrics for Deeper Insights

Using the email subscription app as an example, let's say we are interested in counting the number of successful email subscriptions that are initiated on the app. This is a rather simple metric, but it will give you an idea of what's possible with AppSignal.

To accomplish our task, we can add custom instrumentation to the WelcomeEmailWorker since this Oban worker is responsible for sending the initial user email.

Edit the worker's code as shown below:

# lib/email_subscription_app/workers/welcome_email_worker.ex

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    Appsignal.instrument("welcome_email_worker", fn ->
      email
      |> EmailSender.welcome_email()
      |> Mailer.deliver()
      |> case do
        {:ok, _} ->
          Appsignal.increment_counter("successful_email.subscriptions", 1)
        {:error, reason} ->
          raise "Mailer delivery error: #{inspect(reason)}"
      end
    end)
  end
end

Here's what's going on:

We use Appsignal.instrument/2 to wrap the worker's perform function and create a span (which will be sent over to AppSignal as an event).
Then, we use Appsignal.increment_counter/3 to count the number of successful email deliveries, which we use as the sign of a successful subscription.

Once that is done, we need to set up a custom dashboard to visualize this metric.

Setting up Custom Dashboards for Oban on AppSignal

Start by clicking on the "Add dashboard" button on the left-hand side menu:

Then the "Create a dashboard" button on the next screen:

Now give the new dashboard a title and description:

Adding a Custom Oban Graph

Next, click on the "Add graph" button of your new dashboard:

Then you'll see the fields needed for the custom graph:

Here is a brief outline of what you're looking at in terms of the fields:

a. Graph title - Input a relevant title for the graph.
b. Graph description - Add a description.
c and d. Graph metrics - This one needs a bit more of an explanation. Metric here is the metric we added to our app's counter function. In the email subscription app, this is successful_email.subscriptions (as you can see from the metrics in the screenshot below):

Note: I won't go into the details of all the custom graph and dashboard features available, but you can explore them further in the dashboards documentation.

Finally, when you add a custom graph, you'll start seeing the custom metrics:

And that's it!

Wrapping Up

In this article, we've explored how to integrate and monitor Oban with AppSignal in an Elixir/Phoenix application. We started by setting up Oban for background job processing and then added AppSignal for monitoring.

We discussed common challenges with Oban and how AppSignal's automatic instrumentation can help you gain insights into job execution times, slow jobs, and errors. Additionally, we demonstrated how to create custom metrics and dashboards for more granular monitoring tailored to your application's needs.

By leveraging these tools, you can ensure your background job processing is efficient, reliable, and well-monitored, ultimately leading to a more robust and performant application.

Happy coding!

Advanced Strategies to Deploy Phoenix Applications with Kamal

Roy Tomeij — 2025-07-08T00:00:00+00:00

In the first part of our series, we explored how Kamal simplifies Docker-based deployments while providing a cloud-like developer experience. We covered the basics of containerizing Phoenix applications, configuring Kamal, managing secrets, and implementing a CI/CD pipeline for automated deployments.

Now that we've established a solid foundation for deploying Phoenix applications, it's time to dive deeper into more advanced deployment scenarios that leverage both Kamal's flexibility and the Elixir ecosystem's robust distributed capabilities. These strategies will help you scale your Phoenix applications beyond a single instance and build resilient, production-grade systems.

The Erlang VM (BEAM) was designed from the ground up for building fault-tolerant, distributed systems. When combined with Kamal's multi-role deployment capabilities, we can create sophisticated architectures that maximize the strengths of both technologies. Whether you're running background processing with dedicated worker containers, setting up clustered Elixir nodes, or implementing zero-downtime scaling strategies, Kamal provides the infrastructure orchestration layer while Phoenix and the BEAM provide the application resilience.

In this second part, we'll explore how to:

Configure multi-role deployments to separate web servers from background workers
Establish Elixir clustering across containers for distributed Phoenix applications
Implement advanced monitoring solutions for production visibility

By the end of this article, you'll know how to deploy complex Phoenix applications that can scale horizontally while maintaining the operational simplicity that Kamal offers.

Let's begin by extending our basic deployment to handle multiple roles and establish communication between them.

Multi-Role Deployments with Kamal

One of the most common requirements for growing Phoenix applications is the need to separate concerns between web servers and background workers. This separation allows you to:

Scale web servers and background jobs independently based on their specific resource needs
Isolate potentially resource-intensive background tasks from affecting web request performance
Implement different deployment strategies for each component of your application

Kamal makes this separation straightforward through its multi-role deployment configuration. Let's expand our deployment setup to include dedicated worker containers.

Configuring Worker Roles in Kamal

A web role is responsible for handling user-facing requests — this is your Phoenix Server. A worker role can, for example, be responsible for running background jobs such as sending emails or processing data. There is no limit to what roles you can define with Kamal. However, web is a special role that receives requests from kamal proxy if it is enabled.

Let us update our deploy.yml file to define both web and worker roles:

service: my-app
image: my-user/my-app

servers:
  web:
    hosts:
      - 123.456.789.10 # Your web server IP

  worker:
    hosts:
      # Can be same or different server
      - 123.456.789.10
    # Containers can pass custom environment variables - these will be merged
    # with the common env vars
    env:
      clear:
        ROLE: WORKER
    # Containers can run a different command
    # cmd: /app/bin/my_app worker

# Common configuration for all roles
env:
  secret:
    - SECRET_KEY_BASE
    - DATABASE_URL

This configuration introduces several important concepts. We define both web and worker roles, each with its own servers and commands. Each role starts a separate container with the possibility of overriding the command or environment for that container. Using the env configuration inside the role, we can provide different environment variables for different roles.

Note that Kamal expects there to be a web role (which is the default if you skip the role label and just define the IPs inside the server configuration). You can set a different primary_role in the root configuration.

Modifying Your Phoenix Application for Worker Processes

To take advantage of this multi-role setup, your Phoenix application needs a way to start different processes based on the container's role. This depends largely on what background worker strategy you are using. Let's see an example using the popular background jobs engine Oban.

In this code, we first read the ROLE environment variable to determine the container's role. If the ROLE is "WEB", we start the Phoenix Endpoint to handle HTTP requests and an Oban server without any queues.

If the ROLE is "WORKER", we don't start the Phoenix Endpoint. Instead, we start an Oban server with the default configuration to process background jobs.

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    role = System.get_env("ROLE", "WEB")

    children =
      get_children_for_role(role)
      |> Enum.filter(& &1)

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp common_children() do
    # Common components for all roles
    [
      MyApp.Repo,
      MyApp.JobRepo,
      {Phoenix.PubSub, name: MyApp.PubSub},
      {Cluster.Supervisor,
       [
         Application.get_env(:libcluster, :topologies, []),
         [name: MyApp.ClusterSupervisor]
       ]}
    ]
  end

  defp get_children_for_role("WEB") do
    oban_config_without_queues =
      Application.fetch_env!(:my_app, Oban)
      |> Keyword.merge(queues: [], plugins: [])

    common_children() ++
      [
        MyAppWeb.Endpoint,
        {Oban, oban_config_without_queues}
      ]
  end

  defp get_children_for_role("WORKER") do
    common_children() ++
      [
        {Oban, Application.fetch_env!(:my_app, Oban)}
      ]
  end
end

With this structure, your application will start different components based on the container's role:

Web containers will start the Phoenix Endpoint to handle HTTP requests. They will start an Oban server with no queues, so you can call Oban.insert to enqueue background jobs, but those jobs won't be processed.
Worker containers will not start Phoenix Endpoint but instead start Oban with default configuration (you'll have this in config/config.exs or similar) to handle background jobs.

This approach allows you to share code between roles while maintaining separation of concerns at runtime.

Deploying Multi-Role Applications

With this configuration in place, deploying becomes just as simple as before:

bundle exec kamal deploy

Kamal will automatically deploy both web and worker containers according to the configuration. You can also target specific roles for deployment:

bundle exec kamal deploy --roles=web  # Deploy only web servers
bundle exec kamal deploy --roles=worker  # Deploy only workers

This granular control is especially useful for making role-specific changes without disrupting your entire application.

Elixir Clustering Across Containers

One of Elixir's greatest strengths is its ability to run as a distributed system across multiple nodes. A node is an instance of the Erlang VM running your application. For example, in the above configuration, we started one web server and one worker — each of these is a separate node. While they can work fine in isolation, clustering allows them to coordinate better.

For example, clustering makes global process registration possible, so any node can locate and communicate with a specific process running elsewhere in the cluster. In case of failure, supervision trees can restart processes on different nodes, improving fault tolerance. Clustering also enables distributed task queues, shared ETS/Mnesia tables, and the ability to broadcast events across all nodes — essential for real-time features in modern applications.

When deploying with Kamal, we can take advantage of this capability to create resilient clusters that span multiple containers and even multiple physical servers.

Setting Up `libcluster` for Container Discovery

The libcluster library provides strategies for automatic node discovery and connection. For deployments spanning a single node, we'll use the Cluster.Strategy.Gossip strategy, which works well in containerized environments.

First, add libcluster to your dependencies:

# mix.exs
defp deps do
  [
    # ...existing deps...
    {:libcluster, "~> 3.5"}
  ]
end

Next, configure libcluster in your application supervision tree:

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

defp common_children() do
    [
      {Cluster.Supervisor, [libcluster_topologies(), [name: MyApp.ClusterSupervisor]]},
      # ...existing children...
    ]
  end

  defp libcluster_topologies() do
    # When using releases, this can come from config/runtime.exs using Application.get_env/2
    [
      gossip: [
        strategy: Cluster.Strategy.Gossip,
        config: [
          multicast_addr: System.get_env("CLUSTER_MULTICAST_ADDRESS", "233.252.1.32"),
          port: System.get_env("CLUSTER_PORT", "45892") |> String.to_integer()
        ]
      ]
    ]
  end
end

Creating a Distribution Configuration for Release

If you're using Elixir releases (which is recommended for production), you'll need to properly configure distribution settings in your release configuration:

# rel/env.sh.eex
#!/bin/sh

# Set the release distribution to work on the same node and not require FQDN
export RELEASE_DISTRIBUTION=sname

Updating Kamal Configuration for Node Communication

For containers to communicate properly in a cluster, we need to ensure two things:

The nodes have the same value for the RELEASE_COOKIE environment variable.

You can use any of the methods discussed in Part 1 of this post to expose this variable. Just make sure that you add it to your environment secrets inside config/deploy.yml:

# config/deploy.yml
# ...existing configuration...
env:
  secrets:
    - RELEASE_COOKIE
    # ...other secrets...

The nodes can reach each other.

For this, we don't need to configure anything as long as the containers are on the same node. This is because Kamal automatically creates a Docker network that all nodes join and can communicate with each other.

Once the nodes join the same cluster, you'll unlock a host of benefits that come with it. For example:

If you use Phoenix.PubSub with the PG2 adapter (which is the default), you'll see that PubSub messages are now delivered automatically across nodes. That means you can broadcast from a background job and subscribe to that event inside a LiveView.
Communicate with processes on other nodes simply like you would do if they were on the same node. OTP handles everything else at the BEAM level once the nodes are clustered — no need for Redis, Kafka, or other synchronization tools or message brokers.

Clustering Containers Across Different Nodes

As you've probably already guessed, the Gossip strategy only works across containers on a single node. But that doesn't mean that you'll have to constrain clustered deployments to a single node. We can use other libcluster topologies to enable clustering across nodes, but it'll just need a little more work.

Let's start by configuring libcluster to use the libcluster_postgres strategy. This strategy uses a shared PostgreSQL database to store a list of nodes in the cluster. All nodes listen for and send notifications to a shared Postgres channel. When a node comes online, it begins to broadcast its name in a "heartbeat" message to the channel. All other nodes that receive this message attempt to connect to it.

# config/runtime.exs

# Libcluster is using Postgres for Node discovery
# The library only accepts keyword configs, so the DATABASE_URL has to be
# parsed and put together with the ssl pieces from above.
postgres_config = Ecto.Repo.Supervisor.parse_url(System.fetch_env!("DATABASE_URL"))

libcluster_db_config =
  [port: 5432]
  |> Keyword.merge(postgres_config)
  |> Keyword.take([:hostname, :username, :password, :database, :port])
  |> Keyword.merge(ssl: System.get_env("DATABASE_SSL", "true") == "true")
  |> Keyword.merge(ssl_opts: [cacerts: :public_key.cacerts_get()])
  |> Keyword.merge(parameters: [])
  |> Keyword.merge(channel_name: "my_app_clustering")

config :libcluster,
  topologies: [
    postgres: [
      strategy: LibclusterPostgres.Strategy,
      config: libcluster_db_config
    ]
  ]

The next part is ensuring that the containers can communicate with each other across nodes. Unfortunately, Kamal doesn't (yet) allow exposing custom ports from the servers over the public network.

To enable nodes to connect to each other, Erlang uses a small service called The Erlang Port Mapper Daemon (epmd). It runs on each machine and acts as a name server, mapping node names to their corresponding IP addresses and ports.

So to allow clustering across physical nodes, we'll need to start a separate accessory and proxy the epmd requests to the application servers through that proxy.

First, configure a Traefik proxy as a Kamal accessory inside the configuration.

# config/deploy.yml
accessories:
  traefik:
    service: traefik
    image: traefik:v3.1
    options:
      publish:
        - "6789:6789"
    cmd: "--providers.docker --providers.docker.exposedByDefault=false --entryPoints.epmd.address=:6789 --log.level=INFO"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

To start it, simply run:

bundle exec kamal accessory boot traefik

This will start the accessory on all servers defined in your configuration.

We can use Traefik's labeling system to label our containers so that it forwards all TCP traffic received on the epmd port 6789 on the host to port 6789 of the container.

This way, we start a Traefik proxy on each server to forward epmd requests to the application containers. Each node uses the libcluster_postgres strategy to discover nodes in the cluster. Once libcluster is able to discover nodes and epmd is accessible across all nodes, Erlang is able to establish a cluster.

servers:
  web:
    hosts:
      - 1.2.3.4
    env:
      clear:
        RELEASE_NODE: my_app_web@1.2.3.4
    labels:
      traefik.enable: true
      traefik.tcp.routers.epmd.rule: "ClientIP(`0.0.0.0/0`)"
      traefik.tcp.routers.epmd.priority: 5
      traefik.tcp.routers.epmd.entryPoints: epmd
      traefik.tcp.routers.epmd.service: epmd
      traefik.tcp.services.epmd.loadBalancer.server.port: 6789

The deployment steps don't change. All you need is still only a single command to deploy your apps:

bundle exec kamal deploy

Advanced Monitoring Solutions

In a production environment, visibility into your application's performance and health is crucial. While Kamal makes deployments easy, it's very important to keep monitoring your app for any problems that might occur.

AppSignal provides comprehensive monitoring for Elixir and Phoenix applications. Setting it up with your Kamal deployment is straightforward.

First, add AppSignal to your dependencies:

# mix.exs
defp deps do
  [
    # ...existing deps...
    {:appsignal_phoenix, "~> 2.0"}
  ]
end

Install it inside the app with:

mix appsignal.install YOUR_PUSH_API_KEY

Update your Kamal configuration to inject the AppSignal push API key as a secret:

# config/deploy.yml
# ...existing configuration...

env:
  clear:
    APPSIGNAL_APP_ENV: "production"
  secret:
    - DATABASE_URL
    - APPSIGNAL_PUSH_API_KEY

Check out our full guide to integrate AppSignal into Phoenix.

Logging and Log Management

In addition to application monitoring, AppSignal also makes it easy to store application logs for debugging issues.

Once the AppSignal SDK is integrated, we can configure the Erlang :logger to use the Appsignal.Logger.Handler module as a handler, by calling the Appsignal.Logger.Handler.add/2 function in your Application.start/2 callback:

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    Appsignal.Logger.Handler.add("phoenix")
    # start supervisor and other config...
  end
end

Scaling Strategies

As your application grows, you'll need to scale horizontally to handle increased traffic. Kamal only supports manual scaling out of the box. To scale, manually adjust your deployment configuration:

# config/deploy.yml
servers:
  web:
    hosts:
      - 123.456.789.10
      - 123.456.789.11
      - 123.456.789.12

  worker:
    hosts:
      - 123.456.789.10

This configuration deploys:

3 web containers (1 per server)
1 worker container

To scale up, you simply update the configuration and redeploy (with bundle exec kamal deploy).

While this might look like a downside compared to most cloud service offerings that provide some form of auto-scaling during peak loads, it is important to note that Kamal does not maintain a running daemon. It manages deployments on a push-based model (where updates are pushed to the server on deployments), so auto-scaling is impossible. To deal with this, Kamal advocates over-provisioning for the spikes since the baseline cost (moving from cloud providers to self-hosted servers or budget providers like Hetzner) is several times lower.

Wrapping Up

In this comprehensive guide to advanced Phoenix deployment with Kamal, we've explored how to leverage both technologies to build resilient production systems. By implementing multi-role deployments, Elixir clustering, and advanced monitoring, you now have the tools to deploy even the most complex Phoenix applications with confidence.

The combination of Phoenix's distributed capabilities and Kamal's simple yet powerful deployment model gives you the best of both worlds: the resilience and scalability of the Erlang VM with the operational simplicity of modern containerized deployments.

As you continue to evolve your deployment strategy, remember that simplicity and automation are key. Each enhancement to your deployment process should make your life easier, not more complex. With Kamal and Phoenix, you have the foundation to build systems that scale seamlessly while remaining manageable for small teams.

Happy deploying!

Advanced Ecto for Elixir Monitoring with AppSignal

Roy Tomeij — 2025-07-01T00:00:00+00:00

In our previous article, we explored the basics of monitoring Ecto with AppSignal, covering everything from initial setup to tracking key metrics such as query execution time and resource consumption. We even set up custom instrumentation for database connection pools to gain deeper insights into our application's performance. However, setting up monitoring is just the first step toward maintaining a healthy, high-performing Elixir application. The real power comes from knowing what to do with all that data.

In this second part of our Ecto monitoring series, we'll take your monitoring strategy to the next level. We'll explore how to establish meaningful baseline metrics, set up intelligent alerts that notify you of issues before they affect users, and implement robust exception monitoring to catch database problems early.

Prerequisites

Elixir, the Phoenix framework, and PostgreSQL installed on your development environment.
Some experience with Elixir and the Phoenix framework.
An AppSignal account. If you don't have one ready, sign up for a free trial.
An Elixir/Phoenix app to follow along with. If you don't have one, fork the blog app we'll be using for this tutorial.

Let's get started.

Establishing Baseline Metrics for Ecto

Database monitoring is only valuable as long as you know how to distinguish between "healthy" and "unhealthy" database metrics. Obviously, whether a metric is healthy or not is relative to the application, the hosting environment, and a host of other variables.

All the same, it's important to establish baseline metrics to see whether your database is performing optimally or not.

Why Should You Use Baseline Metrics?

Baseline metrics allow you to define what normal database behavior looks like versus abnormal behavior which could impact your users negatively.

Baseline metrics define expected performance characteristics under normal conditions. They help us answer questions like:

How many database operations per second should we expect?
How are connection pools being utilized when our app is under load?
How long does a query take?

By answering such questions, you will be able to:

Use data-driven optimization - By knowing the healthy ranges under which your app's Ecto database operates, you can set up optimizations backed up by data. For example, if fetching posts from your blog app takes 30 ms, you could reliably set up database indexes that reduce this to, say, 20 ms.
Early problem detection - By knowing what "normal" metrics look like, you can set up an early warning system to alert you to problems affecting Ecto in advance. For example, imagine a query that normally takes 10 ms to execute, which suddenly spikes to 200 ms in execution time. With a baseline already set up, such a spike is flagged as an anomaly that needs to be checked.
Smart resource planning - Knowing how to allocate server resources for your app is very key. Instead of flying blind when it comes to allocating server resources, you can use baseline trends to figure out if your app's resources should be scaled up or down.

Now that we have an idea of what baseline metrics are and why we need them, let's learn how to set some up in the next section.

Collecting Initial Performance Data

Note: We won't go through the process of setting up a new application and adding the AppSignal integration; the previous article has a good walk-through, in case you need a refresher.

Before actually collecting any performance data for our baseline needs, we need to decide what data is important.

In the context of the blog app, the following operations make good examples:

Fetching a post with comments from the database by hitting the GET /posts/:id endpoint.
Creating a post using the POST /posts endpoint.
Something that will put a strain on the database index, like listing all posts (GET /posts).

These examples allow us to collect our initial baseline metrics for Ecto.

Having established the baseline metrics we'll be collecting, the next step is to simulate web traffic that will allow AppSignal to collect the data we need and display it in a dashboard.

Using a Load Testing Tool to Collect Baseline Metrics

We'll use a load testing tool to simulate web traffic to some of the endpoints mentioned above.

There are many load testing tools you can choose from, but for the purposes of this tutorial, we'll go with the k6 load testing tool, an easy-to-use command-line tool for load testing. You can follow this guide to install k6 on your machine.

Once you have k6 ready to go, let's create a simple script that simulates website traffic for one hundred virtual users and fetches a blog post with comments on it.

Baseline Metrics for Fetching a Post With Comments

The blog app's Post context contains the get_post!() function. This fetches a post by id and also preloads any comments on that post:

# lib/blog_app/posts.ex

defmodule BlogApp.Posts do
  ...

  def get_post!(id), do: Repo.get!(Post, id) |> Repo.preload(:comments)

  ...
end

Let's create a simple k6 script that will simulate a hundred virtual users fetching a post:

Note: Create this script in a different folder from your app's folder.

// k6_simulations/script.js

import http from "k6/http";
import { sleep } from "k6";

export const options = {
  vus: 100, // 100 virtual users
  duration: "30s", // How long the test will run
};

export default function () {
  http.get("http://localhost:4000/posts/6"); // We fetch a post with a few comments...
  sleep(1);
}

In the script, we define:

vus - the number of virtual users to simulate
duration - how long the simulation will take (in seconds)
sleep(1) - to simulate real user behavior where users pause between requests, as opposed to continuous requests, which wouldn't be normal.

Then run the script with:

k6 run script.js

After the script runs and you give AppSignal some time to gather the data from Ecto, you should get a dashboard as shown below:

From the graph, we can see that it takes about 30 ms to load the post and another 32 ms to load the associated comments.

Next, let's see how Ecto behaves when we fetch all posts at once using the same one hundred virtual users.

Baseline Metric When the Database Index Is Under Strain

For this one, modify the k6 script as shown below:

// k6_simulations/script.js

import http from "k6/http";
import { sleep } from "k6";

export const options = {
  vus: 100, // 100 virtual users
  duration: "30s", // How long the test will run
};

export default function () {
  http.get("http://localhost:4000/posts"); // We fetch all posts
  sleep(1);
}

Then run it with k6 run script.js, wait, and then view the resulting dashboard on AppSignal:

From the results, we can see Ecto takes around 117 ms to fetch all posts and comments.

Now that we have these metrics, the next question is: "What do we consider to be healthy query ranges for Ecto to compare against?"

Determining Healthy Ecto Ranges and Benchmarks

Determining healthy metrics for Ecto queries is very subjective because no two apps are completely identical in every way, and there are numerous variables that can affect query times.

That said, the following benchmark metrics are worth considering:

Sub-50 ms - For common queries, such as fetching single posts and comments, this is a good benchmark to compare against. Our comparison metric of 30 ms - 32 ms falls within this range, which is good and possibly due to us preloading the comments in the get_post() function.
50 ms - 100 ms - This is considered normal for heavier queries, for example, fetching all posts and associated comments. From the results we got when we load-tested the blog app, we can see that 72 ms is cutting it really close and may indicate there's a need to optimize the query.
Greater than 100 ms - Queries that fall within this range can be considered "very slow." Any query in this range needs immediate optimization because the effect of such a slow query will definitely affect user experience.

Now that we've determined the benchmark metrics we can measure our app's queries against, it would be very convenient if we could get notifications and alerts whenever our app experiences slow queries. AppSignal has an excellent notification and alerting layer that can help us with this.

In the next section, let's learn how to set up alerts and notifications.

Setting up Effective Alerts for Ecto On AppSignal

Having learned about baseline metrics and what to benchmark against, it wouldn't make sense to keep on manually checking dashboards or discovering performance issues after users complain. Instead, we can make use of AppSignal's anomaly detection to notify us when Ecto queries move from normal to abnormal behavior.

In this section, we'll configure alerts that will help you catch database performance problems before they impact your users. We'll set up alerts for slow queries, but the same techniques can be applied for other metrics such as high database connection usage and other critical metrics that could indicate potential issues with your Ecto setup.

For our first example, we'll set up an alert that will trigger whenever an Ecto query takes more than 100 ms to execute.

We will take the following steps:

Setting up a trigger - A "trigger" is what determines if an alert should be raised.
Configuring the trigger - Set up the trigger to fire whenever a metric exceeds pre-determined bounds.
Setting up alert warmup and cooldown - As important as alerts can be, you also don't want them to fire off too many times. Warmups and cooldowns let you control this.
Configuring the alerting service - Here, you get to choose how an alert will be delivered to you. AppSignal gives you a wide choice, from email to SMS alerts, and everything in between.

Setting Up a Trigger

To access triggers on the AppSignal dashboard, go to the left-hand side menu, then locate a menu category called "Anomaly detection," under which you'll find the "Triggers" menu:

Then, click on the "Add a trigger" button, which opens up a dialog like the one shown below:

On the new trigger dialog, we have a few options to set up:

1. Metric type - You can set up a trigger for a variety of metrics: error rates, slow actions, CPU usage, and many more. For our example, we'll go with "slow actions" since we want to get an alert whenever Ecto queries take too long to execute.
2. Namespace - Choose the namespace you want to set up the trigger in. Read more on namespaces in AppSignal's documentation.
3. When to trigger the alert - Here's where you determine the actual conditions under which the trigger will be raised. You can set up a trigger to go off based on the mean, the 90th, or 95th percentile of a metric's measure.
4. Alert warmup and cooldown - These help prevent notification spam. Warmup is the period AppSignal waits before sending the first alert after a trigger condition is met, ensuring the issue persists. Cooldown is the minimum time between subsequent alerts for the same trigger, preventing repeated notifications for ongoing issues.
5. Notification channel - The notification channel to be used. AppSignal gives you a wide choice, including email, SMS, PagerDuty, and others. See more about these in AppSignal's notification settings documentation.

After you've set up the trigger, it will appear in the active triggers list, as shown below:

Now, whenever conditions match the trigger you've set, the trigger fires, and the event that sets off the trigger is listed in the anomalies list, as shown in the screenshot below.

At the same time, you're sent a notification on the notification channel you've set up.

Moving on, let's now turn our attention to monitoring Ecto errors beyond what is captured by AppSignal's default settings.

Monitoring Custom Errors Using AppSignal

By default, AppSignal automatically instruments most of the common Ecto errors, such as non-existent database records. Now, imagine a scenario where you want to track the spam comment submissions within our blog app. How would you handle this? This scenario requires us to set up custom error tracking.

To begin with, we modify the Comment schema to check for spam comments:

defmodule BlogApp.Comments.Comment do
  use Ecto.Schema
  import Ecto.Changeset
  alias BlogApp.Posts.Post

  @spam_phrases ["this is a spam comment", "buy now", "make money fast"]

  schema "comments" do
    field :content, :string
    belongs_to :post, Post

    timestamps(type: :utc_datetime)
  end

  @doc false
  def changeset(comment, attrs) do
    comment
    |> cast(attrs, [:content])
    |> validate_required([:content])
    |> validate_no_spam()
  end

  defp validate_no_spam(changeset) do
    content = get_field(changeset, :content) || ""

    if Enum.any?(@spam_phrases, &String.contains?(String.downcase(content), &1)) do
      changeset
      |> add_error(:content, "Spam detected!")
    else
      changeset
    end
  end
end

The above code uses the validate_no_spam() function to check for spam comments. If a comment matching any of the pre-determined spam comments — "this is a spam comment", "buy now", or "make money fast" — is submitted by a user, a validation error is raised.

Next, we modify the post controller to track spam comments as shown below:

defmodule BlogAppWeb.PostController do
  ...

  @spam_error "Spam detected!"

  ...

  def add_comment(conn, %{"post_id" => post_id, "comment" => comment_params}) do
    post = Posts.get_post!(post_id)

    case Posts.create_comment(post, comment_params) do
      {:ok, _comment} ->
        conn
        |> put_flash(:info, "Comment added successfully.")
        |> redirect(to: ~p"/posts/#{post}")

      {:error, %Ecto.Changeset{} = changeset} ->
        # Track spam attempts
        if changeset.errors[:content] == {@spam_error, []} do
          track_spam_attempt(conn, post_id, comment_params)
        end

        render(conn, :show, post: post, changeset: changeset)
    end
  end

  defp track_spam_attempt(conn, post_id, comment_params) do
    # Get the user's IP
    ip =
      conn.remote_ip
      |> Tuple.to_list()
      |> Enum.join(".")

    # Prepare metadata
    metadata = %{
      content: comment_params["content"],
      post_id: post_id,
      user_ip: ip,
      path: conn.request_path,
      user_agent: get_req_header(conn, "user-agent") |> List.first(),
      timestamp: DateTime.utc_now()
    }

    # Send custom error to AppSignal
    Appsignal.send_error(
      :spam_attempt,
      "Spam comment detected",
      "Content: #{comment_params["content"]}",
      [
        namespace: "spam_detection",
        metadata: metadata,
        tags: ["spam", "user_content"]
      ]
    )

    # Also increment a counter metric
    Appsignal.increment_counter("spam_attempts", 1, metadata)
  end
end

Now with this bit of code, we have the following flow:

User submits a spammy comment (for example, "make money fast").
The schema validation fails via the validate_no_spam() function, and a {:content, {"Spam detected!", []}} error is added.
The controller pattern matches the error and executes track_spam_attempt(), which then triggers a custom event that is sent to AppSignal.

And now, when a user submits a spam comment, this is tracked as an error and displayed on the AppSignal dashboard:

The actual submitted comment is also included:

To go further, you can even set up a custom dashboard to track the number of spam comment submissions — but we won't go into that for now.

Wrapping Up

In this article, we've learned how to establish meaningful baseline metrics to help you distinguish between healthy and problematic database behavior in Ecto. We then configured intelligent alerts using AppSignal that notify you before performance issues impact your users and implemented custom error tracking for spam detection.

With these monitoring strategies in place, you're now equipped to maintain a high-performing Ecto for Elixir application.

Happy coding!

Deploying Phoenix Applications with Kamal

Roy Tomeij — 2025-06-10T00:00:00+00:00

Deploying Phoenix applications in production environments poses unique challenges due to Elixir and the Erlang Virtual Machine (VM).

The ecosystem offers multiple strategies — ranging from releases to mix-based approaches (as detailed in the Phoenix Deployment Guide) — and various platforms employ different methods. While some rely on buildpacks (for example, Heroku and Gigalixir), others use containerization (like Fly). These hosted solutions simplify deployment but often come with a premium cost.

Kamal stands out by providing a cloud-like developer experience while remaining deployable on any infrastructure, from bare metal to cloud VMs. With a focus on simplicity and convention over configuration, it streamlines Docker-based deployments. Although it originated in the Rails ecosystem, its language-agnostic approach makes it an excellent fit for Phoenix applications.

Understanding Kamal

Launched in 2023 by Basecamp (formerly 37signals) and led by David Heinemeier Hansson (DHH), Kamal was created as a straightforward yet powerful deployment tool. Its core principles — strong defaults modeled on “convention over configuration,” an intuitive CLI with clear feedback, and features such as health checks, rollbacks, and zero-downtime deployments — simplify the Docker workflow, from image building to container orchestration. This minimizes operational overhead, letting you concentrate on application development.

When compared to other tools in the Elixir ecosystem, Kamal avoids the infrastructure lock-in and premium costs of Heroku-style solutions while offering a gentler learning curve than Kubernetes. For Phoenix applications, it strikes a balance between managing distributed systems and maintaining an accessible interface that doesn't demand a dedicated DevOps team.

Preparing Your Phoenix Application

Before deploying with Kamal, ensure your Phoenix application is containerized. You may choose to use releases or run mix phx.server directly within the container — both approaches require a Dockerfile to copy your source or release bundle. For releases, run the mix phx.gen.release --docker task (see the Phoenix Release Deployment Guide); otherwise, adjust the Dockerfile from the guide as needed. For secrets management, Kamal’s integration with password managers prevents hardcoding sensitive information.

With these preparations complete, your Phoenix application will be ready for deployment with Kamal, properly containerized and configured for production environments.

Deployment Process

Installing Kamal

After containerizing your application, set up Kamal for deployment. Since Kamal is distributed as a Ruby gem, install Ruby (using version managers like mise-en-place or asdf) and create a Gemfile in your project's root:

bundle init
bundle add kamal
bundle install
bundle exec kamal version

Then initialize Kamal with:

bundle exec kamal init

This generates key configuration files: config/deploy.yml for deployment settings, .kamal/secrets for secret management, and .kamal/hooks for any custom tasks.

Provisioning a Server

To deploy an app with Kamal, you'll need to provision a server. This could be a cloud server (for example, from Hetzner or a Digital Ocean droplet), a bare-metal server, or even a computer on your local network. The only requirement is that it be configured with vanilla Ubuntu and your development machine can SSH to it.

To enable SSH access to the server, copy your SSH public key to ~/.ssh/authorized_keys on the server. If you need to generate a key, you can do that with:

ssh-keygen -t ed25519

Kamal Configuration

The config/deploy.yml file defines the deployment setup: the servers' IPs, container settings, and environment variables. For example:

# Name of your application. Used to uniquely configure containers.
service: my-app

The next part is the container image name. This will be used to upload the image to a container registry, so you'll need something that the registry understands. For example, if you are uploading to the GitHub Container Registry, the image name will be <<org-name>>/<<app-name>>.

# Name of the container image.
image: my-user/my-app

Next, we define the servers the app should be deployed to. If you are only deploying the web server, the servers key will contain a single entry (the web role). For complex apps (e.g., ones that run background workers), it is possible to start multiple containers per app — either on the same server or different servers. Additionally, it is also possible to deploy the same role to multiple servers (for example, when you want multiple web servers behind a load balancer). We'll see the advanced options available in the next post.

For now, let's focus on a single web server setup. We'll need to add the IP of the server we provisioned in the configuration file of the previous step.

# Deploy to these servers.
servers:
  web:
    - 192.168.0.1
  # job:
  #   hosts:
  #     - 192.168.0.1
  #   cmd: bin/jobs

Next, we configure the proxy. Kamal comes with a built-in proxy: the public-facing part of your server, which then forwards all incoming requests to the respective server. The proxy is capable of handling SSL termination and automatic SSL certificate provisioning via Let's Encrypt.

# Enable SSL auto certification via Let's Encrypt.
proxy:
  ssl: true
  host: app.example.com
  # Proxy connects to your container on port 80 by default.
  # app_port: 3000

Next, we configure the container registry that Kamal uses. The container registry is where all built images will be uploaded to and then pulled on the servers. The default is Docker Hub, but many others are supported using the server configuration.

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  # server: registry.digitalocean.com / ghcr.io / ...
  username: my-user
  password:
    - KAMAL_REGISTRY_PASSWORD

Then, we configure the builder that builds the image from the Dockerfile. We can mostly leave our app as-is: the defaults are quite sufficient.

# Configure builder setup.
builder:
  arch: amd64
  # args: ...

Now let's inject environment variables into the containers. Kamal handles them in two stages: clear and secret. Clear variables have a value defined inline in the configuration file. The secrets, on the other hand, only have a name inside the configuration file. Kamal uses .kamal/secrets to read the variable's name and injects it into the containers. We'll learn more about them in the secrets management section of this post.

# Inject ENV variables into containers (secrets come from .kamal/secrets).
#
# env:
#   clear:
#     DB_HOST: 192.168.0.2
#   secret:
#     - RAILS_MASTER_KEY

There are also several other advanced configuration options that we'll explore later on in this and the next post.

Managing Secrets

Kamal uses the .kamal/secrets file to expose sensitive data to containers. A typical structure is:

SECRETS=$(kamal secrets fetch ...)

REGISTRY_PASSWORD=$(kamal secrets extract REGISTRY_PASSWORD $SECRETS)
DB_PASSWORD=$(kamal secrets extract DB_PASSWORD $SECRETS)

kamal secrets fetch supports several adapters including 1Password, LastPass, Bitwarden, among others.

If you don't want to use a secrets manager, it is also possible to expose environment variables as secrets by accessing the environment variable using $NAME inside secrets. For example, the below example exposes a secret named REGISTRY_PASSWORD with the value $KAMAL_REGISTRY_PASSWORD (the value of the environment variable named KAMAL_REGISTRY_PASSWORD):

REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD

Server Bootstrap

If this is the first time you are deploying to a server with Kamal, you'll need to install some tools on the server. Kamal makes this easy through the kamal server bootstrap command. Once config/deploy.yml is ready, just run the bootstrap command: it will install Docker and set up your server to host deployments from Kamal.

Deploying Your Phoenix Application

Now that we know the anatomy of the Kamal configuration and secrets files and our server is ready to handle deployments, let's get back to deploying our application.

A basic Phoenix application mostly needs two secrets: SECRET_KEY_BASE and DATABASE_URL. Depending on your use case, you might need more, but Kamal makes it easy to define and expose as many environment variables as you like. We'll additionally need a password to access the container registry.

The simplest way to expose them to your app is through environment-level secrets. Generate and export the secrets as environment variables.

$ mix phx.gen.secret
REALLY_LONG_SECRET
$ export SECRET_KEY_BASE=REALLY_LONG_SECRET
$ export DATABASE_URL=ecto://USER:PASS@HOST/database
$ export KAMAL_REGISTRY_PASSWORD=SOME_REGISTRY_PASSWORD_OR_ACCESS_TOKEN

Then update .kamal/secrets accordingly:

SECRET_KEY_BASE=$SECRET_KEY_BASE
DATABASE_URL=$DATABASE_URL
REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD

Here's a minimal deploy.yml that makes use of these variables to deploy the application.

All Systems Go!

Finally, we are there! We can deploy the app with a single command:

kamal deploy

This builds the app, pushes it to the registry, and deploys it on our servers. The deployment takes place in a secondary container which goes live only after all health checks pass. If any step of the process fails, the previous app container keeps actively serving all traffic without any interruptions.

Accessing Remote Console

One of the great features of Kamal for Phoenix applications is the ability to access your IEX console in production. You can do this with:

bundle exec kamal app exec --interactive --reuse "/app/bin/my_app remote"

This can be hard to remember, which is where Kamal aliases come in. Inside the deploy.yml file, an aliases entry can be used to define custom commands:

aliases:
  console: app exec --interactive --reuse "/app/bin/my_app remote"
  shell: app exec --interactive --reuse "/bin/sh"
  logs: app logs -f

With this in the config, we get three new commands that can be run from the app folder (on the local/developer machine):

kamal console # Access iex console on the server
kamal shell # Access the shell prompt on the server
kamal logs # Starts tailing application logs from the server

Now let's turn to some more advanced configurations we can do in Phoenix.

Advanced Phoenix-Specific Configurations

We can make our configuration more advanced by automating database migrations and through GitHub Actions / CI/CD pipelines. Let's look at automating migrations first.

Automating Database Migrations with Kamal

Database migrations are one of the most common requirements when deploying applications. With Phoenix, you typically need to run mix ecto.migrate before your application starts serving traffic. While Kamal doesn't have built-in support specifically for Phoenix migrations, we can leverage Docker's entrypoint mechanism to automate this process.

We can create a custom Docker entrypoint script that runs migrations before starting your application.

First, create a file called docker-entrypoint in your project's config directory:

#!/bin/sh -e

# If running the server, run migrations first
if [ "$#" -eq 1 ] && [ "$1" = "/app/bin/server" ]; then
  echo "Running migrations before starting server..."
  /app/bin/migrate
  echo "Migrations completed!"
fi

# Execute the original command
exec "${@}"

This uses the migrate script generated when you bundle the app with mix release. If you are using regular mix commands to run the app, replace them with mix ecto.migrate.

Then update your Dockerfile to use this entrypoint script. In the final stage of your Dockerfile, add:

# Set the entrypoint and default command
ENTRYPOINT ["/app/bin/docker-entrypoint"]
CMD ["/app/bin/server"]

With this setup, whenever Kamal deploys your application, the Docker container will:

Execute the entrypoint script.
The script detects the server starting.
The script runs migrations first.
Then it starts your Phoenix application.

This approach ensures that your database is always up to date before your application starts handling requests. It's particularly valuable in Kamal's zero-downtime deployment process, as the new version of your application will only receive traffic after migrations have successfully completed.

Integration with GitHub Actions or CI/CD Pipelines

The final step in creating a truly modern deployment workflow is to integrate Kamal with your CI/CD pipeline. This integration brings you closest to the convenience of Platform as a Service (PaaS) offerings like Heroku or Fly, allowing your team to focus on writing code while deployments happen automatically in the background.

With CI/CD integration, you can automatically deploy when changes are merged to your main branch (optionally after your tests and linting processes pass to ensure quality). This also creates a consistent, reproducible deployment process.

GitHub Actions provides a straightforward way to implement this workflow.

There are many ways to provide your apps with access to secrets during deployment. The easiest is to store your deployment secrets in GitHub's repository secrets. For our dummy app, this includes:

SECRET_KEY_BASE: Your Phoenix application's secret key
DATABASE_URL: Connection string for your database
KAMAL_REGISTRY_PASSWORD: Password or token for your Docker registry
SSH_PRIVATE_KEY: SSH key for accessing your deployment servers

Then, create a workflow file at .github/workflows/ci.yml with the following content:

name: Deploy

on:
  push:
    branches: [main]

jobs:
  test:
    # Your existing test job
    runs-on: ubuntu-latest
    # ...

  lint:
    # Your existing lint job
    runs-on: ubuntu-latest
    # ...

  deploy:
    needs: [test, lint]
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
    timeout-minutes: 20
    env:
      DOCKER_BUILDKIT: 1
      VERSION: ${{ github.sha }}
      SECRET_KEY_BASE: ${{ secrets.SECRET_KEY_BASE }}
      DATABASE_URL: ${{ secrets.DATABASE_URL }}
      KAMAL_REGISTRY_PASSWORD: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v3
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.1
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Deploy
        run: bundle exec kamal deploy --version=$VERSION

This workflow does the following:

Runs on every push to the main branch.
Waits for tests and linting to pass before deploying.
Sets up all necessary tools (such as Docker, Ruby, and SSH).
Passes your secrets as environment variables to Kamal.
Deploys using the Git commit SHA as the version tag.

The --version=$VERSION flag tells Kamal to tag your Docker image with the Git commit SHA, making each deployment uniquely identifiable and traceable back to a specific commit.

There are several alternatives for managing secrets in this workflow:

Environment Variables (shown above): The simplest approach, which uses GitHub secrets directly.
1Password Integration: If you use 1Password for team secret management, you can update the workflow to install 1Password CLI and modify your .kamal/secrets file to fetch from 1Password.
Other Secret Managers: You can adapt the workflow to use Bitwarden, LastPass, or other supported secret managers.

For larger teams, you might also want to consider setting up staging and production environments with separate workflows triggered by different branches or using GitHub environments for more controlled deployments.

With this CI/CD integration, your development workflow becomes the following:

Develop features in branches.
Open pull requests for review.
Merge approved pull requests to main.
GitHub Actions automatically tests, builds, and deploys your application.

This fully automated pipeline gives you all the convenience of a PaaS while maintaining complete control over your infrastructure and deployment process — the best of both worlds.

Wrapping Up

Whether you're deploying a simple Phoenix application or a complex distributed system, Kamal provides a solid foundation that grows with your needs.

By following the steps outlined in this article, you'll have a modern, efficient deployment workflow that lets you concentrate on what matters most: building great applications with Elixir and Phoenix.

For those interested in exploring further, here are some topics to consider:

Multi-Role Deployments: Set up separate web and background worker containers using Kamal's multi-role support.
Elixir Clustering: Configure distributed Erlang clusters across multiple containers using libcluster.
Advanced Monitoring: Integrate with application performance monitoring tools like AppSignal for comprehensive monitoring of your Phoenix applications.
Automated Scaling: Implement horizontal scaling strategies for handling traffic spikes.
Backup and Disaster Recovery: Establish robust backup procedures for your databases and application state.

We will discuss some of these topics in the next part of this series. Until then, happy coding!

How to Set Up Tracing for Elixir Apps Using AppSignal

Roy Tomeij — 2025-06-03T00:00:00+00:00

Over time, web applications have evolved from simple request/response-based systems into complex, distributed ones with lots of moving parts. If something goes wrong (and you can be sure it will), finding the cause can be nearly impossible. But this need not be the case: enter tracing.

Tracing refers to the process of collecting detailed information about the execution of requests within an application, including function calls, execution time, and other relevant data.

In this article, we'll learn what tracing is all about, including the benefits and challenges of tracing. In addition to this, we'll look at distributed tracing and show you how to set up tracing using AppSignal for a Phoenix LiveView financial news tracker app.

Prerequisites

To follow along with this tutorial, you'll need:

Elixir and the Phoenix framework installed. You can follow this guide to set them up.
An Elixir application. For this tutorial, we'll use the Phoenix LiveView financial news tracker app.
An AppSignal account. If you don't have one, you can sign up for a free trial.
Some experience working with Elixir and the Phoenix framework.

With that done, let's learn what tracing is all about.

Understanding Tracing

Tracing is the process of collecting data during the runtime execution of an application. It gives you visibility into what an individual service (an internal module or function, an external integration, or a combination of all these) does as part of a request cycle. Essentially, the term "tracing" includes both "application tracing" and "distributed tracing".

Application Tracing vs. Distributed Tracing

Application tracing and distributed tracing can be collectively termed as "tracing". However, with application tracing, you follow the execution path of a request within an application or a single service. For example, tracing a user authentication flow that starts and completes within an application without calls to an external system is a good example of an application trace.

But now imagine a distributed system where an app has to interact with external services. The news tracker application which we'll use throughout this article is a perfect example of this.

Here, a request crosses several service boundaries — from the app, through one or more background-job services, to the news-fetching API, then into a cache service that stores the fetched news items, and finally back to the user's browser. Even though this example is simple, tracing such a request flow goes beyond what happens inside the app; it covers every service and system the request touches. This is what we mean by "distributed tracing".

Why We Trace

Without tracing in place, it's difficult to understand what is going on with your app, which ultimately makes it challenging to measure the impact that various issues have on the application.

Here are some reasons why you might want to trace:

Performance analysis - There are many factors that could affect the performance of an Elixir application, such as slow API calls, unoptimized database queries, network latency, and more. But if you have tracing in place, you can pinpoint the issues slowing down your app and fix them.
Helping with debugging - Using the example news tracker app, let's say users submit tickets about not getting news updates on time. Is the problem the API, the database, the news cache, or a combination of all three? Without proper tracing, finding where the issues are coming from can become a real headache for app developers.
Auditing - From a compliance standpoint, tracing can be used to check if your application is leaking any sensitive data (such as usernames or emails) in server logs and other places where this data should not be.

Of course, there are other reasons why we trace, but we'll leave it at that for now. Next, let's see why you should use AppSignal for tracing your Elixir application.

Why Use AppSignal for Elixir?

AppSignal is an excellent application performance management tool with integrations for most modern languages and frameworks. AppSignal offers out-of-the-box integration for pure Elixir and Phoenix apps; for other frameworks and packages, you can add integrations using custom instrumentation, which isn't that difficult to set up.

The Elixir App

The application we'll work with throughout this tutorial is a Phoenix LiveView app that showcases a real-world application. It's a financial news tracker app that:

Fetches financial news from the AlphaVantage API
Manages user-specific news topic preferences
Delivers real-time updates via LiveView
Uses background jobs to fetch data
Implements a caching layer to improve performance

You can see the architecture of the app below:

This architecture is perfect for demonstrating both application and distributed tracing.

In case you haven't done so already, go ahead and fork the app from this Github repository.

Installing the AppSignal Package

Open up mix.exs and add the AppSignal package as shown below:

# mix.exs
...
defp deps do
    [
      ...
      {:appsignal_phoenix, "~> 2.13"}
    ]
end
...

After that, run mix deps.get to add the package to the application, then run the command below to install it:

mix appsignal.install <YOUR API KEY HERE>

After running the installer, you should be prompted for the name of the app:

Validating Push API key: Valid! 🎉
What is your application's name? [email_subscription_app]: Email Subscription App

After that, you'll need to choose the configuration method:

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

Choose the option that suits you best, although the config file option makes it easier to customize the configuration as you wish.

Next, since we are working with Phoenix LiveView, we need to take some extra setup steps.

Adding AppSignal Phoenix Support

Support for Phoenix instrumentation comes in a separate package that depends on the main package we just installed. To add the Phoenix support package, add it to mix.exs:

defmodule FinanceNews.MixProject do
  ...
  defp deps do
    [
      ...
      {:appsignal, "~> 2.8"},
      {:appsignal_phoenix, "~> 2.0"}
    ]
  end
end

Then add the LiveView telemetry integration in the app's application.ex file, like so:

defmodule FinanceNews.Application do

  use Application

  def start(_type, _args) do
    Appsignal.Phoenix.LiveView.attach() # add this line
    children = [
      ...
    ]
    ...
  end
end

Tip: AppSignal offers two different ways to instrument live views, using an automatic handler or through manual instrumentation.

And with that, we have what we need to start tracing our app's requests. But, before we move on to the actual implementation, it's very important we understand the building blocks of tracing and the types of data we want to collect using our tracing setup.

The Building Blocks of Tracing

As we've already mentioned, tracing is about understanding the journey a request takes through a system. That said, a trace is fundamentally made up of several building blocks such as transactions and spans, trace context, sampling, and tags and attributes.

Transactions and Spans

A "transaction" represents a complete unit of work in the application. You can visualize this using the structure of a tree, where the root is the initial request and the branches (spans) are the individual operations that make up that request.

A span contains information like the name of the operation, the start and end times of the operation, a list of events that took place during the operation, and more.

Trace Context

This is a very important concept in distributed tracing, since it defines the relationship between different parts of a transaction across different services and even across system boundaries. For example, and as you will soon see, when our news tracker app makes an API call to fetch the latest news updates, it's a good idea to know what is going on before, during, and after the API call. This is what we mean by "trace context".

The diagram below illustrates this concept using our news tracker app:

Sampling

As much as we want visibility into every part of an application, it would be very costly and probably even negatively impact your app's performance. At some point, you'll have to decide what to trace and what to ignore. This becomes especially important in distributed systems where requests can go through complex transactions and cross many service boundaries. Sampling is the art of choosing representative transactions that will provide the most value to the tracing activity.

Tags and Attributes

Tags and attributes are descriptive pieces of information you use to add context to your traces.

What to Trace vs. What to Ignore

When deciding where to set up traces in your application, understanding what data to collect can be the difference between an effective and ineffective trace implementation.

With that in mind, here are some valuable types of tracing data:

Request/response data - In the example app we're using, having visibility into how various requests and responses are happening is very valuable.
Performance metrics - For example, it would be good to know how long a LiveView render of the latest news articles takes.
Errors - If an error occurs (let's say when fetching news items from the API), it is valuable to know how the error started and progressed.
Custom data - One of the benefits of using a tool like AppSignal is the ability to use custom instrumentation to capture data that's unique to your application.

Now that we know what data we should collect using a tracing setup, let's implement it, starting with basic tracing, before moving on to more advanced tracing methods later.

Implementing Tracing

To get started with tracing, we need to:

Set up a data capture layer using instrumentation.
Send the tracing data to a third-party visualization system.
Analyze the data for further action.

Since we are using the AppSignal library for Phoenix/Elixir, we get the first two without much effort on our part. The last bit is on us completely.

Tracing Web Transactions

In a Phoenix application, web transactions are the entry point of user interactions with our application, and will often be the trigger for subsequent and more advanced transactions later on.

Using our example application, we can break down a web transaction as shown below:

Initial request for information
Controller and LiveView actions
Database queries
Rendering views

To see how these are represented on the AppSignal dashboard, consider the handle_event/3 "save_topics" function below:

# lib/finance_news_web/live/topic_live.ex

defmodule FinanceNewsWeb.TopicLive do
  ...

  def handle_event("save_topics", _, socket) do
    user = socket.assigns.current_user
    topics = MapSet.to_list(socket.assigns.selected_topics)

    case Topics.update_user_topics(user, topics) do
      {:ok, _} ->
        {:noreply, push_navigate(socket, to: ~p"/feed")}

      {:error, _} ->
        {:noreply, assign(socket, error_message: "Failed to save topics. Please try again.")}
    end
  end
end

From this code snippet, you can see:

An interaction with the database to fetch the currently logged-in user
Another interaction with the database to fetch or update the user's selected topics
Rendering the FeedLive LiveView or displaying an error message in case something goes wrong

A request that makes use of this function will be traced and represented on your dashboard in AppSignal, as shown below:

The trace shows:

The name of the event
A breakdown of the sample showing the time taken by the request while interacting with different services in the app. In this case, the Ecto transaction takes the most amount of time compared to the LiveView render.

But what does this really tell us? Or, put another way, how does this information help us?

Analyzing the Results

One use case for such trace data is to help you establish your app's baseline performance. What this means is that, by collecting such data over time, you will soon establish what is acceptable versus what is not in terms of performance.

For example, over time, you might discover topic updates take on average 20-30ms to load, so if you come across trace samples that show loading times of 50ms+, you would know something is amiss. You can then put performance alerts in place, with a setup that uses AppSignal's custom instrumentation, as below:

# lib/finance_news_web/live/topic_live.ex
defmodule FinanceNewsWeb.TopicLive do
  ...
  def handle_event("save_topics", _, socket) do
    Appsignal.instrument("Topics.save_topics", fn ->
      user = socket.assigns.current_user
      topics = MapSet.to_list(socket.assigns.selected_topics)

      case Topics.update_user_topics(user, topics) do
        {:ok, _} ->
          {:noreply, push_navigate(socket, to: ~p"/feed")}

        {:error, _} ->
          {:noreply, assign(socket, error_message: "Failed to save topics. Please try again.")}
      end
    end)
  end
  ...
end

Next, let's look at a more advanced example where we trace an external service.

Tracing External Services

You'll often find external services to be the most unpredictable part of any application. Such services can be integral to your app's functioning, but since you don't have 100% control, you'll find that they can fail, time out, or become slow without warning.

You can see that sourcing the latest finance news items is key to the functioning of our financial news tracker app. Yet this part of the app completely relies on an external service, the Alpha Vantage API.

As the API is important to our app's functioning, let's see how we can implement tracing for such an external service.

Note: AppSignal offers automatic instrumentation for HTTP libraries like Finch (we're using Finch in the news tracker app), so there's no need to add any instrumentation for now. But if you need a custom, more detailed trace, go ahead and use a custom instrumentation.

The results of tracing external service requests are shown below:

Details of the trace are also shown below (note how AppSignal conveniently places such traces under the appropriate "background" namespace):

And that's it!

Wrapping Up

In this article, we learned that tracing goes far beyond simple error tracking. By using various examples related to our financial news tracker application, we've seen how proper instrumentation provides complete request visibility, performance insights, and even deep information on errors.

With tracing in place, you'll have much greater visibility into how requests flow through your system, empowering you to see bottlenecks and optimize the performance and reliability of your Elixir apps. Combine this with the intuitive dashboards provided by AppSignal, and you have everything you need to professionally run an Elixir application in production.

Happy tracing!

An Introduction to Ecto for Elixir Monitoring with AppSignal

Roy Tomeij — 2025-05-13T00:00:00+00:00

Database performance can make or break your Elixir application. While Ecto provides a powerful toolkit for database interactions, understanding how these operations perform in production is critical. Whether you're dealing with slow queries, connection pool issues, or mysterious N+1 problems, the ability to effectively monitor and optimize your database operations can be the difference between a sluggish application and one that delights your users.

In this introductory guide, you'll learn how to implement Ecto monitoring using AppSignal. We'll explore some basic monitoring strategies that will help you identify, diagnose, and resolve database performance issues before they impact your users. By the end of this article, you'll understand:

How to set up Ecto monitoring in your Elixir application
Which key metrics matter most for database performance
How to detect and resolve common Ecto performance bottlenecks
Best practices for setting up alerts and dashboards

Before we get started, let's see what you'll need to follow along with this tutorial.

Prerequisites

An AppSignal account. You can sign up for a free trial if you don't have one.
Elixir, Phoenix, and PostgreSQL installed
A Phoenix application to follow along with. Fork the Phoenix blog app we'll be using for the tutorial if you don't have one ready.
Some experience with Elixir and the Phoenix framework.

What Is Ecto for Elixir?

Ecto is a toolkit that gives you a seamless way to interact with different databases in Elixir applications. You could be wondering why I use the term "toolkit" versus the more common "object-relational mapper" (ORM). Ecto does map database tables into Elixir structs, but more than that, it also includes some distinct features that make it much more than an ORM:

Query - Ecto includes a macro-based domain-specific language (DSL) to compose powerful queries for fetching data from a database.
Schema - The module gives you all you need to build schemas (Elixir structs that map directly to your app's database tables).
Changeset - Ecto's changeset is responsible for ensuring data integrity in your application.
Repo - this is the main interface that wraps around and interacts with your app's database.

Because Ecto is composed of various modules for interacting with databases, there is a larger surface area where things can go wrong, making effective monitoring essential.

Head over to the Ecto documentation to learn more. For now, let's move on and learn how to integrate AppSignal with an Elixir app.

Integrating AppSignal with an Elixir Application

We'll be using a simple Phoenix blog application. Our first step is to add the AppSignal package like so:

# mix.exs

def deps do
  [
    {:appsignal_phoenix, "~> 2.0"}
  ]
end

Run mix deps.get to install the package, followed by the AppSignal package installer command mix appsignal.install <YOUR PUSH API KEY>. This should give you a few options to customize how the package will be installed:

Validating Push API key: Valid! 🎉
What is your application's name? [email_subscription_app]: Blog App

Choose the AppSignal configuration method:

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

Decide what will work for you, but the config file option makes it easier to customize the configuration the way you'd like.

One thing to take note of is that AppSignal will offer automatic instrumentation of Ecto queries as long as your app's OTP name matches what is configured in the AppSignal config file, as shown below:

# config/appsignal.exs

import Config

config :appsignal, :config,
  otp_app: :blog_phoenix, # the otp name here...
  name: "blog_phoenix",
  push_api_key: "YOUR PUSH API KEY",
  env: Mix.env

# config/dev.exs

config :blog_phoenix, BlogPhoenix.Repo,
  ...

And if everything goes as planned, your app should start sending data to AppSignal in a few minutes:

Next up, let's briefly go through the importance of monitoring Ecto.

Why Monitoring Ecto Is Crucial for Application Performance

Just like other parts of your application, your database and the Ecto interface wrapped around it require constant observation to detect issues early and to maintain good performance.

Without monitoring, performance bottlenecks can go unnoticed until they end up directly affecting your app's users. By setting up a well-structured monitoring system for Ecto, you are able to see issues before they grow into serious problems.

But in order to fix the issues that would end up affecting Ecto's performance, you must set up monitoring for the right metrics.

Monitoring Key Ecto Metrics

Focusing on key metrics will give you insights into the efficiency and responsiveness of your database queries, allowing you to identify bottlenecks and optimize database operations.

Below are a few key Ecto metrics that are good candidates for monitoring in AppSignal.

Query Execution Time

Tracking how long queries take to execute is a very important metric. Queries that are not properly optimized will affect the performance of your database.

For example, in our blog application, the query for fetching all blog posts used by the controller's index method is shown below:

# lib/blog_phoenix/blog.ex

defmodule BlogPhoenix.Blog do
  ...

  def list_posts do
    Repo.all(Post)
  end

  ...
end

As you can see, this query is not optimized in any way. To test it out, I seeded the app's database with two thousand posts, and then I made a request for the posts index view.

Let's see how this was instrumented and displayed on the AppSignal dashboard:

As expected, the unoptimized query led to a high-impact event on Ecto, which was captured in detail on the dashboard.

Later on, we'll look at a few ways to optimize queries, but for now, let's examine another metric that warrants monitoring: resource consumption.

Resource Consumption

Although not directly related to Ecto, it's likely your app will be hosted on a server. While it's hosted there, it's possible that whatever might affect the server's resources (such as processor speed, database read/writes, and memory) will also affect Ecto's performance.

For example, something totally unrelated to your app could cause the server to experience high disk read/writes. If your app performs a large number of database transactions at the same time, then it's possible that Ecto's performance will be affected. Being able to see the server's performance metrics provides an extra set of important insights that could help you optimize your application.

Next up is throughput.

Throughput

Throughput refers to the number of transactions completed within a given timeframe. The higher your app's throughput numbers, the better.

In the context of AppSignal monitoring, it's good to note that "throughput" does not show how many transactions or users your app can support as a whole. Rather, it is a snapshot into the number of transactions Ecto can process for a particular event:

So far, the instrumentation for the metrics we have looked at is provided automatically by the AppSignal package. However, there are other metrics that are just as important but which would need some extra work on our part.

Let's look at how to set up custom instrumentation for a particularly interesting metric: database connection pools.

Database Connection Pools

The connection pool size is an important metric to monitor because it directly affects Ecto's performance and therefore the overall performance of your app.

Each database connection consumes a bit of your server's resources (server processor time, memory, and so on). Too many connections will overload your server, directly affecting the performance of your app. But then again, having too few will mean that new requests have to wait for connections to become available, which also affects your app's performance.

As such, it becomes necessary to monitor your app's database pool size to ensure optimal performance.

By default, AppSignal will not offer automatic instrumentation for this metric, but that doesn't mean we cannot set up custom instrumentation for it. Let's do that next.

Setting up Ecto Custom Instrumentation

The first step is to set up a dedicated PoolMonitor Genserver to monitor the database pool size and collect metrics regularly:

# lib/blog_phoenix/pool_monitor.ex

defmodule BlogPhoenix.PoolMonitor do
  use GenServer
  require Logger
  alias Ecto.Adapters.SQL

  @spec start_link(any()) :: GenServer.on_start()
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, %{})
  end

  @impl true
  def init(state) do
    schedule_metrics_collection()
    {:ok, state}
  end

  @impl true
  def handle_info(:collect_metrics, state) do
    _ = collect_pool_metrics()
    schedule_metrics_collection()
    {:noreply, state}
  end

  defp schedule_metrics_collection do
    Process.send_after(self(), :collect_metrics, :timer.seconds(15))
  end

  @spec collect_pool_metrics() :: :ok
  defp collect_pool_metrics do
    pool_size = BlogPhoenix.Repo.config[:pool_size]

    # Get current connection status using a SQL query
    used_connections =
      case SQL.query(BlogPhoenix.Repo, "SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()") do
        {:ok, %{rows: [[count]]}} -> count
        _ -> 0
      end

    # Send metrics to AppSignal
    :ok = Appsignal.set_gauge(
      "database.pool.size_total",
      pool_size,
      %{repo: "BlogPhoenix.Repo"}
    )

    :ok = Appsignal.set_gauge(
      "database.pool.used_connections",
      used_connections,
      %{repo: "BlogPhoenix.Repo"}
    )

    # Calculate and send usage percentage
    usage_percentage = (used_connections / pool_size) * 100
    :ok = Appsignal.set_gauge(
      "database.pool.usage_percentage",
      usage_percentage,
      %{repo: "BlogPhoenix.Repo"}
    )

    Logger.debug("Pool metrics collected: size=#{pool_size}, used=#{used_connections}, usage=#{usage_percentage}%")
    :ok
  end
end

Without explaining everything the code is doing, let's highlight some parts of it:

We make use of Ecto.Adapters.SQL.query/4 to safely execute raw SQL queries using Ecto. This way, we can fetch the database connection pool without interacting with the PostgreSQL database directly, which is the recommended way to interact with the database.
The SQL query uses the PostgreSQL-specific pg_stat_activity to return the number of connection pools. You can read more about the PostgreSQL cumulative statistics system from the documentation.

Next, we ensure this new GenServer is included in the supervision tree:

# lib/blog_phoenix/application.ex

defmodule BlogPhoenix.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      BlogPhoenixWeb.Telemetry,
      BlogPhoenix.Repo,
      BlogPhoenix.PoolMonitor # new Genserver
      ...

      opts = [strategy: :one_for_one, name: BlogPhoenix.Supervisor]
      Supervisor.start_link(children, opts)
  end
  ...
end

With that, we are now ready to visualize the metrics on the AppSignal dashboard.

Setting Up Custom Dashboards

As you might have guessed, since we've built a custom instrumentation, we'll need to set up custom dashboards to view the metrics.

Start by clicking on the "Add dashboard" button on the left-side menu, then the "Create a dashboard" button on the resulting screen:

Then give your dashboard an appropriate name and description:

Once your new custom dashboard is created, the next step is to add custom graphs.

Adding Custom Graphs

Click on the "add new graph" link:

Provide the relevant information for your graph, including a title, description, etc. You can read more in AppSignal's custom metrics documentation.

Let's take a closer look at the metric labeled number 7 on the screenshot. Remember the custom gauges we set up in the PoolMonitor GenServer code above?

...

  :ok = Appsignal.set_gauge(
      "database.pool.size_total",
      pool_size,
      %{repo: "BlogPhoenix.Repo"}
    )
...

We need to pull and visualize these using the custom graph. In my case, I chose database.pool.size_total, which results in the dashboard you can see below:

And that's it!

Wrapping Up

In this article, we learned how to monitor the Elixir database toolkit, Ecto. We've seen the importance of monitoring Ecto and how it affects overall app performance. Additionally, we went through how AppSignal offers automatic monitoring for various metrics. We even learned how to set up custom instrumentation for metrics that aren't automatically covered.

This was more of an introductory tutorial. In part two of this series, we'll dive deeper into how to establish baseline metrics, set up alerts, notifications, and robust exception monitoring for Ecto.

Until then, happy coding!

Advanced Dialyzer Usage in Elixir: Types and Troubleshooting

Roy Tomeij — 2025-04-01T00:00:00+00:00

In our previous post, we covered the basics of setting up and using Dialyzer in your Elixir project. Now, let's dive deeper into advanced type specifications, complex warning patterns, and troubleshooting techniques that will help you make the most of Dialyzer.

Advanced Type Annotations

Advanced type annotations include opaque types, recursive types, and generic types. Let's start by looking at opaque types.

Opaque Types

Opaque types enforce data abstraction by hiding a type's internal structure from external modules. This ensures that any data manipulation occurs solely through a module's public API, thereby safeguarding the integrity of the data.

For example, let's create a Counter module that uses an opaque type to hide its internal structure. This allows us to change the implementation details later without affecting code that uses the module:

defmodule Counter do
  @opaque t :: %__MODULE__{
    value: non_neg_integer(),
    max: pos_integer()
  }
  defstruct [:value, :max]

  @spec new(pos_integer()) :: t()
  def new(max), do: %__MODULE__{value: 0, max: max}

  @spec increment(t()) :: {:ok, t()} | {:error, :max_reached}
  def increment(%__MODULE__{value: value, max: max} = counter) when value < max do
    {:ok, %{counter | value: value + 1}}
  end
  def increment(%__MODULE__{}), do: {:error, :max_reached}
end

By marking the type as @opaque, we ensure that other modules can only interact with our counter through the new/1 and increment/1 functions, preventing direct manipulation of the struct fields.

Recursive Types

Recursive types allow you to define complex, self-referential data structures such as trees or nested expressions. They are particularly useful for modeling hierarchical data and for tasks like evaluating an abstract syntax tree in interpreters or compilers.

Here's an example of using recursive types to define and evaluate a simple arithmetic expression tree:

defmodule AST do
  @type node ::
    {:binary, node(), :+| :- | :* | :/, node()} |
    {:unary, :- | :+, node()} |
    {:number, number()}

  @spec evaluate(node()) :: number()
  def evaluate({:binary, left, op, right}) do
    apply_op(op, evaluate(left), evaluate(right))
  end
  def evaluate({:unary, op, expr}), do: apply_unary(op, evaluate(expr))
  def evaluate({:number, n}), do: n
end

By defining the node() type recursively, we can represent expressions of arbitrary complexity while maintaining type safety. This helps catch errors like malformed expressions at compile-time rather than runtime.

Generic Types

Generic types help create flexible and reusable code by allowing types to be parameterized. This enables you to write functions and structures that can work with various data types while still ensuring type safety. They're particularly valuable when implementing common patterns like result types, option types, or collection operations.

Let's look at a practical example using generic types to create a type-safe result wrapper:

defmodule Result do
  @type t(ok, error) :: {:ok, ok} | {:error, error}

  @spec map(t(a, e), (a -> b)) :: t(b, e) when a: var, b: var, e: var
  def map({:ok, value}, fun), do: {:ok, fun.(value)}
  def map({:error, _} = error, _fun), do: error
end

The t(ok, error) type allows us to specify both the success and error types, making it impossible to mix incompatible values. This pattern is incredibly useful for error handling and data transformation pipelines, ensuring type safety throughout the entire chain of operations.

Advanced Warning Patterns in Elixir

We already examined some common warnings in the first part of this series. Let's examine some of the more complex warning patterns you might encounter, starting with invalid contracts.

1. Invalid Contract

In this example:

defmodule StringHelper do
  @spec valid_format?(String.t()) :: :ok | :error
  def valid_format?(str) do
    pattern = ~r/^[A-Z][a-z]+(-[A-Z][a-z]+)*$/
    Regex.match?(pattern, str)
  end
end

This warning occurs because the function returns a boolean (true or false) from Regex.match?/2, but the typespec promises :ok | :error:

lib/example.ex:17:invalid_contract Invalid type specification for function valid_format?.

Always ensure your typespec matches the actual return value. The correct typespec should be:

@spec valid_format?(String.t()) :: boolean()

2. Function Application Arguments

Let's look at a function application argument:

defmodule Calculator do
  @spec sum(list(integer())) :: integer()
  def sum(numbers), do: Enum.sum(numbers)

  def process() do
    sum(["1", "2", "3"])  # Passing strings instead of integers
  end
end

This error occurs when passing arguments of incorrect types to functions:

lib/calculator.ex:6: The function call will not succeed.
Calculator.sum([<<49>>, <<50>>, <<51>>])
breaks the contract
([integer()]) :: integer()

Always ensure the arguments match the function's typespec. Here, converting strings to integers first would fix the issue.

3. Opaque Type Mismatch

Opaque types are a powerful feature in Elixir, enabling true data abstraction. When you mark a type as @opaque, you tell Dialyzer that the type's internal structure should only be visible within the module that defines it. This is particularly useful for:

Encapsulation: Hiding implementation details from other modules.
Data Integrity: Ensuring data is only modified through your module's API.
API Stability: Changing internal representations without affecting external code.
Security: Preventing sensitive data from being directly manipulated.

Let's look at an example where we create a secure storage module that encrypts data, but a consumer module incorrectly tries to manipulate the encrypted data directly:

defmodule SecureStorage do
  @opaque encrypted_data :: binary()

  @spec encrypt(String.t()) :: encrypted_data()
  def encrypt(data), do: Base.encode64(data)

  # Wrong: Another module trying to manipulate opaque type directly
  defmodule Consumer do
    def process_data(data) do
      SecureStorage.encrypt("secret")
      |> String.split(",")  # Error: Treating opaque type as string
    end
  end
end

Leading to this error:

lib/secure_storage.ex:10:21:call_with_opaque
The call String.split('Elixir.SecureStorage':encrypted_data(),<<44>>) contains an opaque term in 1st argument when terms of different types are expected in these positions}.

Opaque types should only be manipulated by the module that defines them. Instead of allowing direct manipulation, provide public functions to interact with opaque types.

4. Range Errors

Dialyzer detects two types of range errors — missing range and extra range.

Missing range is when a typespec contains a larger range than what a function may return. For example:

defmodule Example do
  @spec ok(boolean()) :: :ok
  def ok(true), do: :ok
  def ok(false), do: :error
end

So you'll see:

lib/example.ex:missing_range
The type specification is missing types returned by function.

Function:
Example.ok/1

Type specification return types:
:ok

Missing from spec:
:error

Similarly, extra range is when the typespec contains a larger range than what a function may return. For example:

defmodule Example do
  @spec error(boolean()) :: :ok | :error
  def error(_book), do: :error
end

You get the error:

lib/example.ex:extra_range
The type specification has too many types for the function.

Function:
Example.error/1

Extra type:
:ok

Success typing:
:error

Tips for Working with Dialyzer in Elixir

Dialyzer can sometimes feel daunting, especially if you are just starting with it. Here are some tips for when you are just beginning your Dialyzer journey:

Start with broader types and narrow them as needed This approach allows you to catch obvious errors without being overly restrictive, giving you room to adapt your types as you understand your codebase better.
Use built-in types like term() sparingly Although term() is very generic, relying on more specific types improves type safety and makes your code self-documenting, helping others understand the expected data shapes.
Consider using TypeCheck alongside Dialyzer for runtime type checking during development Runtime checks with TypeCheck can complement Dialyzer's static analysis, catching errors that might slip through and providing immediate feedback during development.
Remember that Dialyzer is success typing — it only reports errors it's certain about This means Dialyzer might miss some potential issues. It's important to combine its analysis with thorough testing and code reviews for comprehensive error detection.

Troubleshooting Dialyzer

While Dialyzer is a powerful tool, you might encounter some challenges when first integrating it into your workflow. When you encounter Dialyzer warnings that are difficult to understand, there are several strategies you can employ. Let's look at output control first.

Output control

Dialyzer includes verbose information by default. You can control the output level:

mix dialyzer --quiet-with-result # Print only the final results
mix dialyzer --quiet             # No output unless there are errors

Incremental Analysis

For large projects, it can sometimes be helpful to analyze one module at a time. This can be configured inside mix.exs by specifying dialyzer.paths:

# mix.exs
def project do
  [
    dialyzer: [
      paths: ["lib/specific_module.ex"]
    ]
  ]
end

Configuration Tweaks

Dialyzer's behavior can be fine-tuned through various configuration options, particularly around Persistent Lookup Table (PLT) management:

# mix.exs
def project do
  [
    dialyzer: [
      plt_add_deps: :apps_direct,
      plt_add_apps: [:mix, :ex_unit],
      plt_ignore_apps: [:mock]
    ]
  ]
end

Here's what each configuration option does:

PLT Dependencies:
- :plt_add_deps controls how dependencies are added to your PLT:
  - :app_tree (default) - Includes all transitive OTP dependencies
  - :apps_direct - Only includes direct OTP dependencies, so is useful for faster analysis
- :plt_add_apps lets you include additional applications beyond your direct dependencies
- :plt_ignore_apps helps exclude specific applications from analysis
- :plt_apps allows you to specify an exact list of applications, replacing the default list
PLT File Management:
- :plt_core_path defines where PLT files are stored
- :plt_file specifies the PLT file path and options

Pro tip: For team projects, consider storing PLT files in a shared location to speed up analysis across the team.

Next up we'll look at ignoring warnings.

Ignoring Warnings

While it's best to fix all warnings, sometimes you need to suppress specific warnings, especially when migrating a large codebase to use Dialyzer. The ignore_warnings configuration helps manage this:

Generate a list of current warnings in the ignore file format:

mix dialyzer --format ignore_file

Create .dialyzer_ignore.exs with the warnings you want to ignore:

[
  {"lib/example.ex", :call},
  {"lib/secure_storage.ex", :call_with_opaque}
]

Configure Dialyzer to use this file:

# mix.exs
def project do
  [
    dialyzer: [
      ignore_warnings: ".dialyzer_ignore.exs"
    ]
  ]
end

This approach helps you get a passing Dialyzer configuration that you can gradually improve as you fix underlying issues.

And that's that!

Wrapping Up

As we've explored in this deep dive, Dialyzer's advanced features provide powerful tools for maintaining code quality. Through proper use of advanced types, you can achieve better encapsulation with opaque types, model complex data structures using recursive types, and create flexible, reusable code with generic types. These type features form the foundation of robust Elixir applications.

While there's a steep learning curve involved in mastering Dialyzer's advanced features, the investment pays off in more maintainable, better-documented, and more reliable code. Spending time upfront on proper type specifications and understanding Dialyzer's intricacies will save you countless hours of debugging and maintenance in the future.

Happy debugging!

Getting Started with Dialyzer in Elixir

Roy Tomeij — 2025-03-18T00:00:00+00:00

Dialyzer (DIscrepancy AnaLYZer for ERlang programs) is a powerful static analysis tool that helps developers identify potential issues in their Elixir code without executing it. It excels at finding type mismatches, unreachable code, and unnecessary functions through sophisticated flow analysis.

In part one of this two-part series, we'll first get to grips with the basics of Dialyzer. In part two, we'll examine more advanced use cases.

An Introduction to Dialyzer for Elixir

In the world of dynamic languages like Elixir, type-related issues traditionally only surface during runtime, making them harder to catch during development. Dialyzer bridges this gap by analyzing your code's type specifications and usage patterns to detect potential problems before they reach production.

When integrated into your development workflow, Dialyzer provides several key benefits:

Enhanced Code Reliability: Catch type-related bugs early in the development cycle.
Better Documentation: Type specifications serve as living documentation and provide useful hints with IDE integration.
Improved Developer Productivity: Identify issues before they cause runtime errors.
Safer Refactoring: Get immediate feedback about type-related changes.

Setting Up Dialyzer for Elixir

While Dialyzer itself comes with Erlang, the recommended way to use it in Elixir projects is through dialyxir, a mix task that makes Dialyzer more convenient to use.

First, add dialyxir to your project's dependencies in mix.exs:

def deps do
  [
    {:dialyxir, "~> 1.4", only: [:dev, :test], runtime: false},
  ]
end

After adding the dependency, install it by running:

mix deps.get

Now you can run Dialyzer with:

mix dialyzer

On the first run, Dialyzer will build a Persistent Lookup Table (PLT), which contains information about your project and its dependencies. This process might take several minutes, but it's a one-time operation. The PLT will be cached and reused in subsequent runs.

The output will look something like this:

Compiling PLT .dialyzer_plt
Starting Dialyzer
Total errors: 0, Skipped: 0, Unnecessary Skips: 0
done in 0m1.34s
done (passed successfully)

If Dialyzer finds issues, it will display warnings like:

lib/example.ex:2:invalid_contract
The @spec for the function does not match the success typing of the function.

Function:
Example.hello/0

Success typing:
@spec hello() :: :error

Although Dialyzer can infer types from your code, adding explicit type specifications helps it perform a more thorough analysis. We'll explore how to write effective type specifications in the next section.

Types and Type Specifications in Elixir

Type specifications serve multiple purposes in Elixir. Beyond documentation, they provide Dialyzer with the information needed to perform sophisticated flow analysis. As someone who's worked extensively with Elixir in production, I've found that well-designed type specifications:

Act as executable documentation
Enable early bug detection
Make refactoring safer
Improve IDE integration

Adding Typespecs to Your Elixir Code

Let's start with the basics and progress to more complex scenarios. There are two main type attributes you'll use frequently:

@type name :: <<type scpecification>> defines a type with the specified name.
@spec function_name(argument_types) :: return_type defines a function specification.

There are several built-in types like any(), none(), atom(), map(), and integer().

It is also possible to compose types using list(type), nonempty_list(type), etc.

You can use types defined in other modules as well. It is quite common for data structures to define their types named t. In fact, several Elixir data structures already define types that can be used in your custom typespecs, like String.t() and Range.t().

Finally, we can use almost any literal as a type that restricts the allowed values to be like the literal. For example, 1, 1..10, {:ok, type}.

We'll cover advanced attributes to create other special types in the second and final part of this series.

For now, here's an example of using some typespecs in practice:

defmodule UserAccount do
  @type status :: :active | :inactive | :suspended
  @type role :: :admin | :user

  @spec create_user(String.t(), status(), role()) :: {:ok, map()} | {:error, String.t()}
  def create_user(name, status, role) do
    # Implementation
  end

  @spec is_active?(status()) :: boolean()
  def is_active?(status), do: status == :active
end

In this example:

We define custom types status and role using union types
The create_user function specification ensures that arguments are of the correct types
Dialyzer will catch type mismatches, like passing a number instead of a string for a name

We'll explore more advanced type attributes like @opaque in the next post.

Understanding Common Dialyzer Warnings

Now that we have Dialyzer set up and running, let's explore the most common warnings you'll encounter and learn how to address them effectively.

1. Match Errors

Dialyzer detects unused pattern matches by analyzing your types:

defmodule Example do
  def ok() do
    unmatched(:ok)
  end

  defp unmatched(:ok), do: :ok

  defp unmatched(:error), do: :error
end

The pattern can never match the type.
Pattern:
:error
Type:
:ok

2. Redundant Guard Clauses

This warning shows Dialyzer's ability to detect impossible conditions:

defmodule Example do
  def to_string(%{} = a) when is_binary(a) do
    inspect(a)
  end
end

# lib/example.ex:2:31:guard_fail The guard clause can never succeed.

Here, we're pattern matching on %{} (a map) but then using an is_binary/1 guard — these conditions can never be true simultaneously since a value cannot be both a map and a binary. Such contradictions often indicate logical errors in your code's flow control. Remove the contradictory guard or fix the pattern matching based on your intended behavior.

3. Unmatched Returns

This warning is particularly important for maintaining robust error handling:

defmodule Example do
  def ok() do
    :rand.uniform(100)
    |> validate()

    :ok
  end

  defp validate(n) when n < 50, do: :ok
  defp validate(n) when n > 50, do: {:error, "too big"}
end

lib/example.ex:unmatched_return
The expression produces a value of type:

:ok | {:error, <<_::56>>}

but this value is unmatched.

Dialyzer has detected that the function returns a union type (:ok | {:error, binary()}) that we are ignoring. In Elixir, it's a best practice to handle all possible return values, especially errors.

To fix this, you should:

Pattern match on the result and handle both cases, or
Use with expressions for cleaner error handling, or
Explicitly discard the result using _ = if that's truly intended

4. No Local Return

This indicates that your function never returns normally — it either raises an exception or runs indefinitely:

defmodule Example do
  def ok() do
    Enum.each([1, 2, 3], fn _ -> raise "error" end)
  end
end

lib/example.ex:no_return
The created anonymous function only terminates with explicit exception.

This is common in placeholder code, but in production, you should ensure your functions have proper return paths.

If the function is meant to be raised, specify no_return() in the typespec:

@spec do_something() :: no_return()

In addition to obvious no-returns, Dialyzer often also raises this as a side-effect of other errors that lead to a guaranteed exception.

For example, the following code tries to pass age as a string when it's declared type is an integer, leading to a Dialyzer no_return error as a side-effect of the call error.

defmodule User do
  @type t :: %__MODULE__{name: String.t(), age: integer()}
  defstruct [:name, :age]

  @spec get_age(User.t()) :: integer()
  def get_age(user), do: user.age
end

defmodule Example do
  def do_something() do
    User.get_age(%User{name: "User", age: "20"})
  end
end

lib/example.ex:10:7:no_return Function do_something/0 has no local return.
lib/example.ex:11:18:call The function call get_age will not succeed.

This can sometimes also happen when a library has incorrect typespecs or complex types that Dialyzer has trouble inferring.

In such cases, it can be desirable to skip Dialyzer warnings. This can be achieved by using a special @dialyzer attribute to disable warnings, specifying a tuple with the warning type and the fuction/arity.

defmodule Example do
  @dialyzer {:no_return, do_something: 0}

  def do_something() do
    User.get_age(%User{name: "User", age: "20"})
  end
end

And that's it for this part of our two-part series!

Wrapping Up

Getting started with Dialyzer might seem daunting at first, but the benefits are worth the initial setup time. As you've seen, it can catch many common issues before they reach production.

If you already have a large project without Dialyzer, the trick is to start small. First:

Add Dialyzer to your development dependencies
Begin with a single module
Add type specifications gradually

In our next and final part of this series, we'll dive deeper into advanced type specifications, explore sophisticated troubleshooting techniques, and learn how to handle complex scenarios that you might encounter as your codebase grows.

We'll also look at how to effectively configure Dialyzer for larger projects and establish team-wide practices for maintaining type specifications.

Until next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Absinthe for Elixir Monitoring with AppSignal

Roy Tomeij — 2025-03-04T00:00:00+00:00

Absinthe is a popular GraphQL toolkit for building robust APIs in Elixir. Monitoring such APIs is essential to ensure performance, detect bottlenecks, and handle errors effectively. AppSignal offers a seamless way to monitor and gain insights into your Absinthe-powered GraphQL APIs, enabling you to keep applications performant and reliable.

In this post, we’ll explore how to leverage AppSignal’s automatic instrumentation and custom metrics for Absinthe, along with strategies to monitor errors, track subscriptions, and enhance performance through actionable metrics.

Automatic Absinthe for Elixir Instrumentation with AppSignal

Setting up AppSignal for your Elixir application is straightforward. You can follow the official setup guide to install the AppSignal package and configure it with your application key. Once integrated, AppSignal automatically instruments your Absinthe GraphQL API, giving you insights into the execution time and performance of individual queries and mutations.

Here's an example of how your AppSignal performance report might look, showcasing query performance breakdown:

The sample breakdown above shows that GraphQL took 45 ms—this duration covers both parsing the query and encoding the results. Ecto took 37 ms, with most of that time spent executing the database query.

Let's check out the Event Timeline now:

As you can see, this provides a breakdown of all the Ecto queries performed during the resolution of this query. With this instrumentation, you can identify slow queries, measure throughput, and understand which resolvers might need optimization—all without adding any manual instrumentation.

Handling Errors and Exceptions in Absinthe with AppSignal

In GraphQL APIs, errors can occur at multiple points: within resolvers, due to validation issues, or from an underlying data source. AppSignal helps capture and report these errors along with contextual information, making debugging efficient.

Resolver Exceptions and Validation Errors

When an error occurs in a resolver, AppSignal captures it and provides details about the query that caused the issue, the resolver, and the stack trace.

For example, if a single field on a GraphQL API has an error, it will be reported on the "Errors" page on AppSignal. Here's an example of such an error:

Clicking on "Inspect latest sample" takes you to the full sample details, including:

A backtrace that can help us pinpoint where the error originated.
The full GraphQL query and variables (if any), and a lot of other information that can be useful to reproduce that issue.

Tracking Custom Metrics

While automatic instrumentation provides valuable insights into the performance of your Absinthe API, tracking custom metrics can offer even deeper visibility into specific areas of your application. Custom metrics allow you to define and collect data points unique to your business logic or performance needs. AppSignal makes it easy to track these metrics in your Absinthe GraphQL API.

Defining Custom Metrics

To start tracking custom metrics, you'll need to define the specific aspects of your API that you want to measure. This could include:

Resolver Performance: Measure how long specific resolvers take to execute.
Query Complexity: Track the number of fields requested in a query or the depth of the query.
Subscription Events: Monitor events related to GraphQL subscriptions, such as connection counts and active subscriptions.

Implementing Custom Metrics

You can implement custom metrics using AppSignal's built-in APIs. Here’s a simple example, tracking the execution time of a resolver:

defmodule MyAppWeb.Resolvers.PostResolver do
  def get_pots(_, %{id: id}, _) do
    start_time = System.monotonic_time()

    case Blog.get_post(id) do
      nil ->
        Appsignal.increment_counter("post.not_found")
        {:error, "Post not found"}

      post ->
        duration = System.monotonic_time() - start_time
        Appsignal.set_gauge("post.fetch_time", duration)
        {:ok, post}
    end
  end
end

Once you've defined and implemented your custom metrics, you can view them in your AppSignal dashboard, which includes various graphs and charts. These graphs and charts are crucial for identifying trends over time, such as increasing fetch times or spikes in 'post not found' errors.

Monitoring Absinthe Subscriptions

Subscriptions in Absinthe allow for real-time updates, typically used in chat or notification systems. Monitoring these subscriptions is crucial to ensure they perform well throughout their lifecycle.

Example: Monitoring Subscription Performance

A typical Absinthe subscription might look like this:

subscription :comment_added do
  arg :post_id, non_null(:id)

  config fn args, _ ->
    {:ok, topic: args.post_id}
  end

  trigger :comment_added, topic: fn %{post_id: post_id} -> "post:#{post_id}" end

  resolve fn comment, _, _ ->
    {:ok, comment}
  end
end

Similar to custom metrics monitoring, we can track subscription activation time, latency during updates, total active subscriptions at any given time, and any errors that occur within these subscriptions, giving you full visibility into their behavior.

Here’s how to add basic instrumentation to measure the time required to set up and resolve the subscription.

field :comment_added, :comment do
  arg(:post_id, non_null(:id))

  config(fn %{post_id: id} = _args, %{context: context} ->
    Appsignal.instrument("subscription.config", "Setting up subscription", fn ->
      {:ok, topic: "post:#{id}"}
    end)
  end)

  resolve(fn comment, _, _ ->
    Appsignal.instrument("subscription.resolve", "Resolving subscription", fn ->
      {:ok, comment}
    end)
  end)
end

And that's it for our whistle-stop tour of Absinthe monitoring using AppSignal!

Wrapping Up

Monitoring GraphQL APIs with AppSignal enables Elixir developers to maintain high performance, quickly diagnose errors, and leverage custom metrics for business-critical operations. Whether you're tracking query execution, debugging resolver errors, or ensuring efficient subscription handling, AppSignal equips you with the necessary tools to build and maintain robust Absinthe applications.

If you are building complex APIs with Absinthe, ensure you integrate AppSignal to enhance your observability and performance monitoring. For further reading, explore AppSignal's Absinthe documentation and this four-part series on building scalable GraphQL APIs with Absinthe.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Building a Distributed Rate Limiter in Elixir with HashRing

Roy Tomeij — 2025-02-04T00:00:00+00:00

Communication between processes in the Erlang VM happens through message passing. In fact, all communication across processes occurs through the same mechanism. All you need is an address and you can send a message to it, with the lower level building blocks being send and receive. This is true even if that address is on another node in a cluster of Elixir nodes that are aware of each other. In fact, it works in exactly the same way.

There’s no special RPC syntax, no special error handling. In fact, you can write your code without knowing whether a message is destined for another node or process running on the same node. In an example send(address, message) the address could be a local pid, “via tuple”, or a pid from another node. It could be a registered name combined with a node in a tuple like {node, MyProcess}. send will figure out what to do. This is also true when using the higher level versions GenServer.cast/2 and GenServer.call/3.

It might not be obvious at first glance, but this is a superpower. Let’s take a simple example of some code that runs on a single node and then upgrade it to share state across a cluster of nodes. To follow along, all you’ll need is Elixir installed.

Note: This blog post is an extended version of an example from my Code BEAM Mexico keynote.

A Basic In-Memory Rate Limiter

defmodule RateLimiter do
  def check(ip) do
    Hammer.check_rate(ip, 60_000, 10)
  end
end

In my example, I’m wrapping the Hammer rate limiting library, although you could use any other library or roll your own. With our RateLimiter module we can now lock down the endpoints in our imaginary web app and protect our service. We’re very strict, only allowing 10 requests per minute per IP.

But as we scale up and add more nodes, this basic rate limiting module doesn’t work anymore because it’s not sharing state across the nodes. The same IP is now able to make more than 10 requests per minute, as the requests are randomly assigned to different nodes by the load balancer.

There are, of course, tons of ways of solving this problem. Hammer itself comes with an adapter for Redis that allows sharing state across multiple nodes. Don't get me wrong — Redis is a great piece of software, but you might not need it. Maybe there’s a way to solve this, using Elixir and nothing else.

Interlude: Connecting Nodes

To try this out on your own machine, you’ll need to connect two Elixir nodes together. To get started, assuming you’ve got Elixir installed locally, you’ll want to start iex but with a special flag:

iex --name a@127.0.0.1

Then, in a second terminal/tab:

iex --name b@127.0.0.1

Now, in either terminal, type in Node.list and verify that it’s an empty list, and then Node.ping(:”x@127.0.0.1”) replacing x with the name of the other node — the one you're not currently using. You should see a :pong response. Now try typing in Node.list again and verify that it shows the other node in the list. That’s it, you’ve made your first cluster!

Note that if you get a :pang, you may need to run the command epmd in your terminal first, to start the service that connects the nodes.

Hash Rings

The first tool we’ll need is a mechanism for associating keys with nodes in a cluster. I’m a big fan of Discord’s HashRing library so we’ll use that, but there are alternative implementations out there. This is based on an ingenious idea from the ‘90s around returning consistent look-up results given the same outputs. Basically, we can add all of our node names to the ring, and when we look up keys we can rely on getting the same result every time (assuming the node names in the ring are the same). Even when they change, we should see minimal changes in our look-ups:

iex(b@127.0.0.1)4> alias ExHashRing.Ring
ExHashRing.Ring
iex(b@127.0.0.1)5> {:ok, ring} = Ring.start_link()
{:ok, #PID<0.279.0>}

# The keys you add to the ring can be anything, but for our purposes
# we're interested in node names.
iex(b@127.0.0.1)6> Ring.add_node(ring, "a@127.0.0.1")
{:ok, [{"a@127.0.0.1", 512}]}
iex(b@127.0.0.1)7> Ring.add_node(ring, "b@127.0.0.1")
{:ok, [{"b@127.0.0.1", 512}, {"a@127.0.0.1", 512}]}
iex(b@127.0.0.1)16> Ring.add_node(ring, "c@127.0.0.1")
{:ok, [{"c@127.0.0.1", 512}, {"b@127.0.0.1", 512}, {"a@127.0.0.1", 512}]}

# Now that they're in there, we can take it for a spin. Let's try checking
# some different look ups.
iex(b@127.0.0.1)18> Ring.find_node(ring, "result1")
{:ok, "c@127.0.0.1"}
iex(b@127.0.0.1)19> Ring.find_node(ring, "result2")
{:ok, "a@127.0.0.1"}
iex(b@127.0.0.1)21> Ring.find_node(ring, "result4")
{:ok, "b@127.0.0.1"}

# Okay, but what if we remove a node?
iex(b@127.0.0.1)22> Ring.remove_node(ring,"c@127.0.0.1")
{:ok, [{"b@127.0.0.1", 512}, {"a@127.0.0.1", 512}]}
iex(b@127.0.0.1)23> Ring.find_node(ring, "result4")
{:ok, "b@127.0.0.1"}
iex(b@127.0.0.1)24> Ring.find_node(ring, "result1")
{:ok, "a@127.0.0.1"}

# As you can see, result4 still results in node b even though the
# memberships changed, and result1 has been moved over to a.
# But what if we add the node c back?
iex(b@127.0.0.1)25> Ring.add_node(ring, "c@127.0.0.1")
{:ok, [{"c@127.0.0.1", 512}, {"b@127.0.0.1", 512}, {"a@127.0.0.1", 512}]}
iex(b@127.0.0.1)26> Ring.find_node(ring, "result1")
{:ok, "c@127.0.0.1"}
# result1 is now back on c. It will always go to c, those are the
# behaviors we can rely on from the hash ring.

Okay! Now let’s hook into OTP to grab all the node names and add them to the ring, and additionally, use some events to keep the ring up to date as nodes connect and disconnect. You can use this code alongside manually connecting and disconnecting nodes from the terminal like I demonstrated above. However, in a production environment, you may want to use a library like libcluster to manage your cluster more efficiently.

First of all, add a HashRing to your application.ex like this {ExHashRing.Ring, name: DistributionRing}. Then add the following module to your app:

defmodule Distribution do
  use GenServer

  require Logger

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl GenServer
  def init(_opts) do
    :net_kernel.monitor_nodes(true)

    for node <- [Node.self() | Node.list()] do
      ExHashRing.Ring.add_node(DistributionRing, node)
    end

    {:ok, []}
  end

  @impl GenServer
  def handle_info({:nodeup, node}, state) do
    Logger.info("Node #{node} connected")
    ExHashRing.Ring.add_node(DistributionRing, node)
    {:noreply, state}
  end

  def handle_info({:nodedown, node}, state) do
    Logger.info("Node #{node} disconnected")
    ExHashRing.Ring.remove_node(DistributionRing, node)
    {:noreply, state}
  end
end

This is a GenServer responsible for maintaining that hash ring, adding and removing nodes as they connect and disconnect. Go ahead and try it out right now if you want. You should see the log messages show up as you add and remove nodes with Node.ping. Pretty neat.

Now we have everything we need to upgrade our simple rate limiting module to share data across the cluster.

Distributed Rate Limiter

Let’s do this in two steps. First, we need to turn our basic rate limiting module into a GenServer so that there’s a process on each node that can be reached from the other nodes. We’re doing it in two steps to really hammer home the difference between the version that only works locally and the version that returns consistent results across the cluster.

defmodule RateLimiter do
  use GenServer

  # Public interface
  def check(ip) do
    check_internal(ip)
  end

  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def init(_opts) do
    {:ok, []}
  end

  # Internal implementation
  def handle_call({:check, ip}, _from, state) do
    {:reply, check_internal(ip), state}
  end

  defp check_internal(ip) do
    Hammer.check_rate(ip, 60_000, 10)
  end
end

Make sure you add that process to your application children, like {RateLimiter, []}. Then let’s upgrade:

defmodule RateLimiter do
  use GenServer

  require Logger

  # Public interface
  def check(ip) do
    {:ok, node} = ExHashRing.Ring.find_node(DistributionRing, ip)
    GenServer.call({__MODULE__, node}, {:check, ip})
  catch
    :exit, reason ->
      Logger.warning("Tried to check rate limit but failed", reason: inspect(reason))

      # As a fallback, do a local check. It's better than nothing!
      check_internal(ip)
  end

  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def init(_opts) do
    {:ok, []}
  end

  # Internal implementation
  def handle_call({:check, ip}, _from, state) do
    {:reply, check_internal(ip), state}
  end

  defp check_internal(ip) do
    Hammer.check_rate(ip, 60_000, 10)
  end
end

We’re taking a laissez-faire approach to error handling here overall. This code is just something to get you started, but I do think it’s worth having that catch in the check function. When GenServer.call/3 fails it exits and crashes the caller, this would, for example, happen where it’s not able to reach the process on the other node. Because networks are not reliable, this will happen. Fortunately, short interval rate limiting is a use case that doesn’t demand absolutely perfect results. We can very easily fall back to just checking locally instead, and at worst, a few extra requests may need to be made. As long as the network is mostly stable, we’ll be fine.

So there it is. With those two magical lines, looking up the node and then passing the node to GenServer.call/3, we took code that ran fully locally, to code that now transparently routes requests to the proper nodes in the cluster. Given a specific IP address, the lookup will always happen on the node that is responsible for that IP address, and even if nodes are going up or down it is likely to be handled by the same node anyway!

A Word of Warning

I do want to emphasize that this isn’t necessarily the “right” tool for every use case. In my opinion, it works well for short-lived rate limiting because it’s ok if we’re a little bit wrong sometimes. It’s also ok that we lose state on app restarts. The rate limit quotas reset every minute anyway!

Wrapping Up

If you’re concerned about routing these rate limit lookups through GenServers, remember that they can easily handle tens of thousands of messages per second. And if you need more than that, you’ll probably need to look at something more robust than this! But for the majority of apps that need multiple nodes, maybe for resiliency and nothing else, this can be a great alternative to Redis or other solutions, helping you keep your stack lean and not worry about maintaining extra services.

To summarize: you get the semantics of just calling code locally, transparent routing across your cluster, and you don’t even need to add another service to maintain. And remember, with 3 nodes, every 3 lookups on average will happen locally, amortizing your overall network latency overhead!

You don’t need to limit yourself to rate limiting either, there are tons of use cases where you can apply this pattern. Why not add some simple ephemeral caches? Or how about some high cardinality metrics/event reporting? Your options are limitless, so why not bring a little bit of distributed Elixir magic into your app.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Tracking Errors in Tesla with AppSignal for Elixir

Roy Tomeij — 2025-01-21T00:00:00+00:00

While Elixir provides built-in libraries (like HTTPoison) for making HTTP requests, the Tesla library has gained popularity due to its simplicity, ease of use, and extensibility. Tesla offers a clean and composable way to define requests, handle responses, and customize client behavior.

Even so, whenever you're working with external APIs, errors are inevitable. Network issues, server downtime, rate limiting, and unexpected responses can all lead to errors in your application. As such, it’s great to have a dependable error tracking mechanism in place to quickly identify, diagnose, and resolve issues.

AppSignal is a powerful application monitoring and error tracking solution that seamlessly integrates with Elixir apps and libraries such as Tesla.

In this article, we'll explore how to effectively track and handle Tesla errors in your app using AppSignal. We'll start by understanding the basics of Tesla and its key features. Then we'll set up a simple weather forecasting application that integrates Tesla with AppSignal. Along the way, we'll discuss common error scenarios, implement error handling techniques, and showcase how to leverage AppSignal's intuitive dashboard to monitor and analyze errors.

Pre-requisites

First, ensure you have the following in place:

An AppSignal account. Don't have one? Sign up for a free trial.
A WeatherAPI account to fetch weather data.
Some experience working with Elixir and the Phoenix framework.
An Elixir/Phoenix application to work with. You can fork the weather forecast app which we'll use throughout this tutorial.

And with that, let's get started!

Brief Overview of Tesla for Elixir

Tesla is a modular Elixir HTTP client library that you can use to build HTTP clients with flexible middleware configurations. This middleware configuration allows you to extend the behavior of your HTTP client, with features such as:

Authentication - With Tesla, you can configure basic or token-based authentication schemes, or even define your own custom scheme.
Error handling - You can configure the middleware to intercept request errors.
Caching and rate limits - You can set up custom caching rules and API call rate limits to reduce the number of API calls your client makes.

Introducing Our Weather App Built with Phoenix

The application we'll use to demonstrate Tesla's features is a simple weather forecasting app built with Phoenix. It will feature a home page with a form input where a user can enter the name of a city. Upon the form submission, the user is redirected to another page that will display a three-day weather forecast. The weather forecast data will be fetched from the Weather API service using a Tesla HTTP client.

Although we won't go through a step-by-step walk-through of building the weather forecast Phoenix app, I will highlight some of the app's key features, especially how Tesla fits in and where some potential issues could come from.

Below is the summarized shell output of the tree command on the lib directory, which contains the most important parts of the app for our discussion:

-> lib
├── weather_forecast
│   ├── ...
│   └── weather_api.ex
├── weather_forecast.ex
├── weather_forecast_web
│   ├── components
│   │   ├── ...
│   ├── controllers
│   │   ├── ...
│   │   ├── weather_controller.ex
│   │   ├── weather_html
│   │   │   ├── forecast.html.heex
│   │   │   └── index.html.heex
│   │   └── weather_html.ex
│   ├── ...
└── weather_forecast_web.ex

API client - The API client found in the lib/weather_forecast/weather_api.ex file uses Tesla to make and parse HTTP requests from the weatherapi.com API.
Weather controller - Contains two actions, index and forecast. The index action renders a form input for entering a city name. The forecast action takes the city parameter and calls the API client to fetch the weather forecast, which responds with the weather forecast data or an error.
The views - The index view displays the form input where a user will enter a city name and be redirected to the forecast view upon form submission. There, a three-day weather forecast or an error will be displayed.

Over the coming sections, we'll use this app to instrument Tesla API requests, errors, and more. For now, let's see how we can add AppSignal and Tesla to the app.

Adding AppSignal and Tesla to the Weather App

Open up the mix.exs file and add the lines shown below:

# mix.exs
...
    defp deps do
        [
            {:tesla, "~> 1.12"},
            {:hackney, "~> 1.20"},
            {:appsignal, "~> 2.8"}
        ]
    end
...

You might be wondering what Hackney is and how it fits in here. While Tesla is a great choice for wrapping HTTP requests intuitively, an adapter is still needed to handle the actual job of opening connections, sending requests, parsing responses, and so forth, which is what Hackney does. That said, Hackney isn't the only adapter; there are other adapters like Finch and Gun that you can use.

With that out of the way, run mix deps.get to fetch the packages, then mix appsignal.install <YOUR APPSIGNAL API KEY> to run the AppSignal interactive package installer, which will ask you to choose:

The name of your application
The configuration method (an environment variable or a configuration file).

You'll get an output similar to the one shown below if the process runs without a hitch.

Validating Push API key: Valid! 🎉
What is your application's name? [weather_app]: weather_tesla_app

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

What is your preferred configuration method? [1]: 1
Writing config file config/appsignal.exs: Success!
Linking config to config/config.exs: Success!
Activating dev environment: Success!
Activating prod environment: Success!
...
AppSignal installed! 🎉

Once done, you should have AppSignal installed, and your app will start sending data to the AppSignal dashboard.

Configuring Tesla

However, we need to add a few configurations to Tesla and Hackney to complete the installation.

To begin with, add the Tesla.Middleware.Telemetry which plugs into the app's built-in telemetry. This needs to be the first middleware in the API client, as shown below:

# lib/weather_forecast/weather_api.ex

defmodule WeatherForecast.WeatherApi do
  use Tesla

  plug Tesla.Middleware.Telemetry  # This line should be first
  # ..then the rest of the middleware follow

  ...
end

But before we continue, it's important to highlight how this use of middleware makes Tesla unique among Elixir HTTP clients.

The Concept of “Composable Middleware” in Tesla

Tesla uses the concept of "composable middleware" similar to the Plug router, which means:

With Tesla, you can define a middleware chain where each middleware is executed in the order it appears for each HTTP request. For example, the Tesla.Middleware.Telemetry middleware is first in the chain in the API client and, as such, will be executed first.
It's heavily inspired by Plug, which makes it easy to use different middleware as you wish.
You have access to several built-in middleware, such as the JSON middleware, the Logger middleware, and others. At the same time, you can define your own custom middleware to implement custom logic on requests.

It's crucial to execute the Tesla.Middleware.Telemetry middleware first, to ensure that:

It records the start of an event before another middleware processes that event.
It captures an event as it is before another middleware processes that event.

Let's now complete the configuration by adding the Hackney configuration to dev.exs:

...

config :tesla, adapter: Tesla.Adapter.Hackney
...

With everything configured, fire up the app and test it out a bit. You should start to see request data on your AppSignal dashboard, as shown below:

Everything looks great so far! Let's now use AppSignal's powerful custom instrumentation features to get more information on Tesla's requests.

Instrumenting Tesla with AppSignal

With the custom instrumentation provided by AppSignal, we can add informational tags to Tesla's requests to give us more insight into what is actually going on under the hood.

Modify the API client's get_forecast/1 function as shown below:

# lib/weather_forecast/weather_api.ex

defmodule WeatherForecast.WeatherApi do
  use Tesla

  plug Tesla.Middleware.Telemetry
  ...

  def get_forecast(city) do
    Appsignal.instrument("Weather API - Get Forecast", "WeatherAPI Forecast for #{city}", fn ->
      client()
      |> get("/forecast.json", query: [q: city, days: 3])
      |> case do
        {:ok, %{status: 200, body: body}} -> {:ok, parse_forecast(body)}
        {:ok, %{status: status, body: body}} ->
          IO.inspect(body, label: "API error response")
          {:error, "API request failed with status #{status}"}
        {:error, error} -> {:error, error}
      end
    end)
  end

  ...
end

An AppSignal.instrument/2-3 function wraps the code where we define it in a span. In the case of the weather forecast app, this is the get_forecast/1 function. When this code is executed, the span captures useful information such as execution time and any errors that might occur.

The screenshots below show how such instrumented requests are captured on the AppSignal dashboard:

And if you click on the link shown, you get even more information on the request:

Tesla Errors

Whenever you work with an HTTP client library like Tesla, you can expect some of the errors listed below:

Rate-limiting errors
Authentication errors
Network errors
Response errors

And more.

Let's pick one of these and show how to custom instrument it using AppSignal.

Instrumenting Tesla Authentication Errors with AppSignal

Different APIs will require different authentication mechanisms. For the WeatherAPI.com API, authentication is accomplished through an API key.

To simulate an authentication error, I'm going to regenerate the API key, but still use the old API key with the weather forecast app. Then I'll modify the API client's get_forecast/1 function, as shown below:

defmodule WeatherForecast.WeatherApi do
  ...

    def get_forecast(city) do
        Appsignal.instrument("Weather API - Get Forecast", fn span ->
        Appsignal.Span.set_sample_data(span, "params", %{city: city})

        client()
        |> get("/forecast.json", query: [q: city, days: 3])
        |> case do
            {:ok, %{status: 200, body: body}} ->
            {:ok, parse_forecast(body)}
            {:ok, %{status: status, body: body}} ->
            error_message = "API request failed with status #{status}"
            stacktrace = get_application_stacktrace()
            Appsignal.send_error(%RuntimeError{message: error_message},
                                stacktrace,
                                fn error_span ->
                                    Appsignal.Span.set_sample_data(error_span, "error", %{
                                    status: status,
                                    body: body,
                                    city: city
                                    })
                                end)
            {:error, error_message}
            {:error, %Tesla.Error{reason: reason} = error} ->
            error_message = "API request failed: #{inspect(reason)}"
            stacktrace = get_application_stacktrace()
            Appsignal.send_error(error,
                                stacktrace,
                                fn error_span ->
                                    Appsignal.Span.set_sample_data(error_span, "error", %{
                                    reason: reason,
                                    city: city
                                    })
                                end)
            {:error, error_message}
        end
        end)
    end

    defp get_application_stacktrace do
        {_, full_stacktrace} = Process.info(self(), :current_stacktrace)
            Enum.filter(full_stacktrace, fn {module, _function, _arity, _location} ->
                to_string(module) =~ "WeatherForecast"
        end)
    end
end

Here's a breakdown of what's going on:

We first wrap the get_forecast/1 function with AppSignal.instrument/2, then define a new span using AppSignal.Span.set_sample_data/3, with the city name as the sample data.
A request is then made to the weather forecast API. If an error occurs (in this case, the authentication error that we expect), the error is created with all the information we need and then forwarded to the app's AppSignal dashboard.
The private get_application_stacktrace function gets the full stack trace of the current process. Since this can include a bunch of entries from other Phoenix/Elixir modules, we filter for those that contain the string "WeatherForecast" as this is what we are really interested in. This filtered stack trace is what gets sent to AppSignal.

You can see the dashboard results below:

As well as the error details:

And, finally, the stack trace sample:

And that's that!

Wrapping Up

In this article, we explored how the Elixir HTTP client library Tesla works.

We learned how to install and use Tesla in a Phoenix application and then installed AppSignal to instrument Tesla requests. We custom-instrumented requests to dig deeper into stack traces and the time taken for API calls. With this information, you are now better equipped to build more effective performance profiling for your Elixir apps.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

AppSignal’s Top 5 Elixir Posts in 2024

Roy Tomeij — 2024-12-17T00:00:00+00:00

As 2024 comes to a close, we’re thrilled to highlight our top five Elixir articles that captured the most attention this year!

Season's Greetings ❆ ⛄

Have a wonderful festive break, and we'll see you in the new year!

If you haven’t already, don’t forget to subscribe to our Elixir Alchemy newsletter, so you never miss an upcoming post.

Distributed Phoenix: Deployment and Scaling

Roy Tomeij — 2024-12-10T00:00:00+00:00

In part one of this series, we managed distributed state using GenServers. This provided a foundation for understanding some core concepts in distributed Phoenix applications.

Now, we turn our focus to deployment and scaling strategies. As your application evolves to meet growing demands, knowing how to scale horizontally, maintain high availability, and monitor distributed components becomes crucial.

This post will walk you through best practices for deploying and scaling Phoenix applications in a distributed setup, ensuring they are performant and resilient.

Distributed Deployment Strategies

Deploying a Phoenix application in a distributed environment introduces some unique challenges and opportunities that are less common to traditional web stacks. Thanks to the underlying power of the BEAM VM, Elixir applications offer native clustering capabilities.

When deployed correctly, a distributed Phoenix app becomes more than a collection of stateless servers — it turns into a unified system where instances can communicate directly, share state, and respond gracefully to network failures.

Let’s explore two common deployment strategies for Phoenix apps: single-node and multi-node deployment.

Single-Node Deployment

In smaller setups, it’s possible to run your Phoenix app as a single instance. You can package the app either as a mix release (recommended for all deployments, as it packages the app into a single directory that includes everything you need to run the app), or simply run it with mix phx.server with the correct environment variables.

This is common for small-scale deployments where high availability isn’t critical and the load can be handled by a single server. The pros are that it's a simple deployment with minimal overhead and is easy to manage. But your web server is also now a single point of failure. Though unsuitable for large-scale distributed systems, single-node deployments are a good starting point before introducing more complexity.

Deployment Across Multiple Nodes

A node in the context of a Phoenix (or Elixir) application refers to a single instance of the BEAM VM, capable of running the Phoenix app and connecting with other nodes to form a cluster.

Each node can share its state and manage processes independently, but can also interact with other nodes when deployed together.

To deploy a Phoenix application across multiple nodes:

Create Multiple Instances: Use tools like Docker, Kubernetes, or your preferred cloud provider (like AWS EC2 or DigitalOcean Droplets) to create multiple instances of your Phoenix application. Each instance will act as an independent node.
Set Up Clustering: Configure the nodes to communicate with each other by setting the RELEASE_DISTRIBUTION and RELEASE_NODE environment variables. For example:
```
RELEASE_DISTRIBUTION=name
RELEASE_NODE=app_name@host
```

Each node should be configured with a unique name (like app_name@host1, app_name@host2, etc.), allowing them to recognize each other within the cluster.

Load Balancing: Place a load balancer in front of nodes to distribute incoming requests evenly. On AWS, for example, you can use an Elastic Load Balancer (ELB) and configure it to forward requests to nodes. In the load balancer setup, specify the DNS or IP of each node and configure health checks to detect if any node goes down.

Testing the Cluster: Once the nodes are up, test that they recognize each other. You can use :observer.start() (or connect via iex shell) to inspect the network status of nodes and verify they’re communicating as expected. This can be done by running and checking the return value. The command will return :pong if the node is reachable and :pang otherwise.

iex> Node.ping(:"app_name@host2")
:pong

With clustering, Phoenix applications can benefit from shared state and inter-node communication. Scenarios like real-time messaging, shared PubSub broadcasts, and distributed task management (e.g., using Horde) are enabled in a clustered setup.

Pros:

Native support for process distribution across nodes.
Enables the use of libraries like Horde for distributed task management.
Stateful GenServers and ETS tables can be shared or replicated between nodes.

Cons:

Requires attention to network partitions (nodes becoming temporarily unreachable).
Inter-node communication adds complexity — issues like split-brain scenarios must be handled.

Scaling a Distributed Phoenix Application

Horizontal scaling is often the go-to strategy for distributed applications. An external user's first point of contact with your app is usually the web layer. Scaling this layer is easy — add more instances of your Phoenix app behind a load balancer to handle increased traffic. To further distribute load, use CDNs to offload static asset delivery.

Before adding more web servers, scaling processes allows an application to handle more load within a single instance by distributing work across lightweight processes. The most common way to achieve this is using background job processing. Using libraries like Oban, an application can spawn lightweight processes to handle background tasks. These processes are isolated and managed independently by the BEAM scheduler, improving concurrency without increasing web layer load.

Finally, for apps that significantly scale, your database needs to keep up with new connections from the scaled-up web and background job processes. The easiest way to scale a database is through vertical scaling — keep adding more CPU power and memory to the database server.

An alternative to this is to have read replicas to distribute read-heavy workloads from the primary database server.

Your setup on the database side will depend a lot on the database (Postgres, MySQL, etc.) and the cloud provider. On the app side, Ecto can be configured with multiple repositories to leverage replicas for reading queries.

Considerations for High Availability

High Availability (HA) ensures that your application remains operational even in the face of failure, minimizing downtime and disruptions. Phoenix applications, powered by the BEAM VM, are particularly well-suited for building HA systems due to native support for fault tolerance, process isolation, and clustering.

Let’s explore strategies for ensuring high availability in Phoenix apps, along with specific tools and techniques tailored for Elixir-based architectures.

Clustering and Node Redundancy

A core feature of the BEAM ecosystem is clustering, where multiple nodes work together to act as a single, logical system.

In a Phoenix app, clustering ensures that instances (nodes) share state and responsibilities dynamically. When clustered, nodes can pass messages between processes on different instances, ensuring tasks or state can failover seamlessly if a node goes down.

Phoenix.PubSub benefits from clustering, enabling nodes to broadcast real-time updates (like notifications) across the cluster without additional complexity.

Load Balancing and Failover

Load balancing ensures that traffic is evenly distributed across all available nodes, while failover mechanisms detect unhealthy nodes and redirect requests to healthy instances. Use NGINX or HAProxy to route HTTP traffic between multiple Phoenix nodes. If using WebSockets, enable sticky sessions to maintain persistent connections between users and the same server. In a Kubernetes environment, use Service objects or Ingress controllers to manage traffic between pod instances.

Set health checks to monitor node availability, removing unresponsive nodes from the load balancer pool.

Use Erlang’s node monitoring features to detect when nodes leave or join the cluster and adjust load distribution accordingly.

Fault Tolerance with Process Isolation

The BEAM VM’s design philosophy revolves around process isolation, meaning that failure in one part of the system does not bring down the entire app. This makes it easier to build systems that gracefully recover from failures.

Every Phoenix app is backed by supervision trees, where supervisors restart crashed processes automatically. In complex apps (e.g., background job workers or live dashboards), GenServers can isolate logic into independent processes that recover individually when something goes wrong.

Zero-Downtime Deployments

In highly available systems, every deployment needs to occur without any downtime, especially when serving global users around the clock.

Here are a few strategies for zero-downtime deployments:

Blue-Green Deployments: Run two identical environments (blue and green) and switch traffic between them seamlessly after the deployment.
Rolling Deployments: Update nodes incrementally, keeping some instances online while others are upgraded.
Hot Code Upgrades: With Distillery or Releases, you can deploy changes without restarting your application (though this requires some additional planning).

Monitoring and Maintenance

No distributed system is truly complete without robust monitoring and proactive maintenance. Ensuring high performance and quick resolution of issues requires tracking key metrics, receiving alerts on failures, and understanding application behavior in real time.

Phoenix applications, with their modular design and reliance on the BEAM VM, offer a wealth of telemetry options.

This section explores monitoring techniques and best practices for keeping a distributed Phoenix app healthy and resilient over time.

Phoenix for Elixir Application Monitoring with Telemetry and Tracing

Monitoring starts with gaining visibility into the core operations of your application. Phoenix provides out-of-the-box support for Telemetry, enabling developers to track key metrics such as response times, database query performance, and request rates.

Phoenix emits Telemetry events for things like HTTP requests, database interactions, and process life cycles. These metrics allow you to pinpoint performance bottlenecks and detect abnormal behavior.

Tools like AppSignal seamlessly integrate with Phoenix and Ecto, providing dashboards for tracking application performance, error rates, and system health.

Error Tracking and Alerts

Even with the best architecture, errors are inevitable in production. A good error-tracking system ensures that you are alerted to issues as soon as they occur, allowing for quick resolution and minimal impact on users.

AppSignal captures errors, including crashes, unhandled exceptions, database timeouts, and more! Here's an example of how an Ecto error looks in AppSignal:

You can also configure alerts based on error frequency or severity to be notified about critical issues in real time.

This can be really useful. For example, a surge of 500 Internal Server Error responses can indicate an overloaded node or an unexpected database issue.

Tracking Metrics Across Nodes

Distributed Phoenix applications require centralized monitoring to aggregate data from all nodes. Without this, diagnosing issues across multiple instances becomes difficult. Some key node metrics to track include:

Node Uptime: Track how long each node has been online and watch for frequent restarts.
Memory and Process Utilization: Monitor the number of BEAM processes and memory consumption on each node.
PubSub Metrics: Track the number of connected WebSocket clients and broadcast messages across the cluster.

Wrapping Up

In this post, we explored key strategies for deploying and scaling distributed Phoenix applications. We’ve covered the essential practices needed for building resilient, scalable systems. With monitoring tools and proactive maintenance, you can ensure your app is ready to meet future demands.

Together, parts one and two provide a comprehensive guide to developing, deploying, and maintaining distributed Phoenix applications.

Mastering these techniques will empower you to deliver robust systems capable of handling real-world challenges.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Track Errors in Oban for Elixir Using AppSignal

Roy Tomeij — 2024-11-12T00:00:00+00:00

When developing an Elixir app, you'll often need to handle tasks in a way that does not interrupt the normal user request-response cycle.

Tasks like sending emails are great examples of jobs that should be delegated to a capable background job processing service. In the Elixir ecosystem, Oban is one such background job processing library.

In this article, we'll learn what Oban is, how it works, and how to instrument it using AppSignal.

Prerequisites

To follow along with this tutorial, I recommend you have the following:

An AppSignal account - you can sign up for a free trial.
An Elixir app. In my case, I am using a Phoenix LiveView app with a landing page where a user can subscribe to an email newsletter with a couple of background jobs to schedule and send out emails. You can go ahead and fork the app.
The AppSignal package added and configured for your app.
Elixir, Phoenix, and PostgreSQL installed on your local development environment.
Some basic experience of working with Elixir and Phoenix.

And with that, we can get started.

Introducing Oban

Oban is a stable and high-performing background job processing library for Elixir applications. Unlike other job processing systems that might require their own infrastructure to run background jobs, such as RabbitMQ and Sidekiq (which needs a key-value store like Redis), Oban uses your app's existing PostgreSQL or SQLite database.

With Oban, you can define, enqueue, and process jobs in the background. Another important thing to note is that Oban jobs are persistent, which means that even if something unexpected happens in the server environment, jobs will be retried for you.

Installing and Configuring Oban in Elixir

First, add Oban to your app's mix.exs like so:

# mix.exs

defmodule EmailSubscriptionApp.MixProject do
  ...

  defp deps do
    [
      ...
      {:oban, "~> 2.17"}
    ]
  end

  ...
end

Then run mix deps.get in the terminal to install the package and its dependencies.

Before moving on, it's important to cover the database your app is using since it will determine how you configure Oban.

If you've already configured PostgreSQL for your app, the Postgres package is likely already installed, and you'll not have to add it manually. However, if your app uses SQLite, you'll have to add the EctoSQLite3 package to mix.exs and configure it accordingly. In this article, the featured app uses PostgreSQL for the database, and Oban is installed to run on top of that.

Next, generate a migration file that will create all the database tables Oban needs to run and keep tabs on job queues.

mix ecto.gen.migration add_oban_jobs_table

Then modify the generated file to look like this:

# priv/repo/migrations/timestamp_add_oban_jobs_table.exs

defmodule EmailSubscriptionApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migration.up(version: 12)
  end

  def down do
    Oban.Migration.down(version: 1)
  end
end

Run the migration with mix ecto.migrate.

With that done, you next need to configure Oban to run with the Postgres engine by adding the following lines to config.exs:

# config/config.exs

config :email_subscription_app, Oban,
  engine: Oban.Engines.Basic,
  queues: [default: 10],
  repo: EmailSubscriptionApp.Repo

It's also recommended that you prevent Oban from running jobs in testing mode by adding the lines below to text.exs:

# config/text.exs

config :email_subscription_app, Oban, testing: :inline

Finally, we'll need to add Oban to the app's list of supervised children since they run as isolated supervision trees. To do this, add Oban to the app supervisor like so:

defmodule EmailSubscriptionApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      ...
      EmailSubscriptionApp.Repo,
      {Oban, Application.fetch_env!(:email_subscription_app, Oban)},
      ...
  end
  ...
end

Then, to finish up the installation and configuration, open up a new interactive Elixir shell with iex -S mix. Run the command Oban.config() to return an output similar to the one below:

iex(1)> Oban.config()
%Oban.Config{
  dispatch_cooldown: 5,
  engine: Oban.Engines.Basic,
  get_dynamic_repo: nil,
  insert_trigger: true,
  log: false,
  name: Oban,
  node: "...",
  notifier: {Oban.Notifiers.Postgres, []},
  peer: {Oban.Peers.Postgres, []},
  plugins: [],
  prefix: "public",
  queues: [default: [limit: 10]],
  repo: EmailSubscriptionApp.Repo,
  shutdown_grace_period: 15000,
  stage_interval: 1000,
  testing: :disabled
}

Before moving on to defining a few jobs and instrumenting them using AppSignal, let's touch on how instrumentation actually works.

Automatic Instrumentation of Oban Using AppSignal

When you added the AppSignal package to your app, Oban instrumentation was automatically added. The AppSignal package provides instrumentation for Oban workers and will also collect metrics on how background jobs are performing.

With that in mind, let's write a few jobs and see how they appear on the AppSignal dashboard.

Defining the First Background Job

Our practice Phoenix application is all about receiving a user's email in an input on the homepage, then sending that user a series of emails (starting with a welcome email that is sent immediately when the user subscribes, a follow-up email sent ten minutes later, and then a final email sent thirty minutes later).

Let's start by defining the background job that will trigger the first email.

# lib/email_subscription_app/workers/welcome_email_worker.ex

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default

  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()

    :ok
  end
end

Let's dig into this example code a bit to understand how Oban workers work. This information will prove helpful when troubleshooting job errors.

You have the module definition — in this case, the WelcomeEmailWorker — then one or more job-performing functions, usually called perform/1.

The perform/1 function receives an Oban.Job struct containing the specific job's arguments which, in our case, is the email key.

Even though we haven't shown it in this example code, Oban also allows for job scheduling using the schedule_at and schedule_in options.

And like we mentioned before, AppSignal automatically instruments Oban and displays some default dashboards, as you can see in the examples below.

Here are our Oban workers listed on AppSignal:

And details of a worker's instrumentation:

Now we've installed Oban, defined a simple background worker job, instrumented Oban with AppSignal, and visualized the output. Let's turn to when things go wrong with Oban and how we can use AppSignal to help us track errors.

Oban Errors

Although Elixir systems are well-known for their fault tolerance, errors and bugs can affect them, resulting in a poor user experience if not addressed. In this section, we'll look at some common errors that could affect your Oban workers, as well as how you can instrument errors with AppSignal and see them on your dashboard.

Many different errors could affect an Oban worker, including:

Database connection errors - Since Oban depends on accessing a PostgreSQL or SQLite 3 database, any connection issues between your app and its database will definitely affect any workers you have defined.
External service failures - For example, if there is an issue that affects emails being sent through a third-party email service provider in an email subscription app, the background jobs we've defined could fail to send emails, which would result in Oban job errors. Further, if you call third-party API services from your app, any rate-limiting efforts by the API service provider could result in connected Oban jobs failing to complete and resulting in errors.
Job timeout errors - These errors occur if a job's actual execution time exceeds the configured time.
Job queue congestion errors - These happen when there are too many jobs in the queue, which could result in delays in job execution.

Instrumenting Oban Errors Using AppSignal for Elixir

One of the major benefits of AppSignal is that most common errors are automatically instrumented for you. For example, see the code snippet below:

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()
    |> case do
      {:ok, _} -> :ok
      {:error, reason} ->
        raise "Mailer delivery error: #{inspect(reason)}"
  end
end

If an error occurs within the perform/1 function, AppSignal will catch it and give you a dashboard view with details of the error, as you can see in the screenshot below:

Before wrapping up this article, let's take a look at the error details AppSignal shows you in the dashboard:

Error message - The actual error message associated with the raised error.
Backtrace - The backtrace gives you fine-grained context on what happens before an error occurs in your app.
Parameters - AppSignal also shows you any parameters involved when the error occurred.
Deploy - Finally, you also get information on whether the error occurred during a deployment or not.

And that's that!

Wrapping Up

In this article, we looked at the background job processing package Oban. We learned how to install it, configure background workers, simulate an error, and instrument errors using AppSignal. I recommend using this information as a foundation for using Oban and AppSignal in your Elixir apps.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Managing Distributed State with GenServers in Phoenix and Elixir

Roy Tomeij — 2024-10-29T00:00:00+00:00

Phoenix and Elixir are designed at their core to build real-time, fault-tolerant applications. With its elegant syntax and the robustness of the Erlang VM, Elixir is an ideal candidate for tackling the challenges of distributed state management.

This two-part series will guide Phoenix/Elixir developers through the intricacies of working with Phoenix in a distributed setup.

The focus of this first post will be on GenServers — the key components behind state management in distributed systems. We will explore how GenServers can be leveraged to maintain state across nodes in a Phoenix application, ensuring data consistency and fault tolerance. We’ll break down the complexities of GenServers and distributed state management into manageable and understandable segments.

Overview of Our Phoenix Application

Let's focus on distributed state management in the context of a CRUD API. Imagine you have a popular API service built with Phoenix that handles a large number of requests from various clients. To prevent abuse and ensure fair usage, you want to implement a rate limiter that restricts the number of requests a client can make within a specific timeframe, such as 100 requests per minute.

You can implement this simply using the "token bucket" strategy. In this strategy, a bucket is maintained for each client that is filled with tokens at a constant rate. Each API call consumes a single token from the bucket and is blocked if there are no tokens available.

In the next section, we will set up a token bucket for a single-node scenario using a GenServer.

A Single-Node Rate Limiter Implementation Using GenServer for Elixir

GenServers are a cornerstone within the Elixir ecosystem for managing state and encapsulating functionality. They are the go-to structure for maintaining state within an application because they can hold state throughout their lifecycle and be interacted with asynchronously.

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  # tokens per second
  @rate 1
  # maximum tokens in the bucket
  @bucket_size 10

  def start_link([]), do: GenServer.start_link(__MODULE__, nil, name: __MODULE__)

  @impl true
  def init(nil), do: {:ok, %{buckets: %{}}}

  def allow?(client_id) do
    GenServer.call(__MODULE__, {:allow?, client_id})
  end

  @impl true
  def handle_call({:allow?, client_id}, _from, state) do
    current_time = System.monotonic_time(:second)
    {reply, updated_bucket} =
      state.buckets
      |> get_or_init_bucket(client_id)
      |> update_bucket(current_time)
      |> maybe_consume_token(current_time)

    state = put_in(state, [:buckets, client_id], updated_bucket)
    {:reply, reply, state}
  end

  defp maybe_consume_token(%{tokens: tokens} = bucket, current_time)
       when tokens >= 1 do
    updated_bucket = %{bucket | tokens: tokens - 1, last_checked: current_time}
    {true, updated_bucket}
  end

  defp maybe_consume_token(bucket, _current_time), do: {false, bucket}

  defp get_or_init_bucket(buckets, client_id) do
    Map.get(buckets, client_id, %{
      tokens: @bucket_size,
      last_checked: current_time
    })
  end

  defp update_bucket(bucket, current_time) do
    # Add tokens based on the last time the bucket was checked
    time_passed = current_time - bucket.last_checked
    new_tokens = time_passed * @rate
    updated_tokens = min(bucket.tokens + new_tokens, @bucket_size)
    %{bucket | tokens: updated_tokens, last_checked: current_time}
  end
end

This works well in a single-node setup because the GenServer holds the token state for the entire application across web requests. We can add it to the application's supervision tree to start the API rate limiter automatically:

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Other Children...
      MyApp.TokenBucketRateLimiter
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # ...
end

The first time allow? is called with a new client ID, it initializes the bucket to 10 tokens, and then each allow? with the same client ID consumes one token per request. Eventually, the eleventh request in the same second is rejected because there are no more tokens. If the client tries again after a while, the requests are allowed again.

iex> 1..11 |> Enum.map(fn _i -> MyApp.TokenBucketRateLimiter.allow?("some client id") end)
[true, true, true, true, true, true, true, true, true, true, false]

iex> Process.sleep(1000)
:ok
iex> MyApp.TokenBucketRateLimiter.allow?("some client id")
true

But as the app scales and we deploy it across multiple nodes (e.g., in a Kubernetes cluster or across different regions), we face the challenge of coordinating rate limiting across all nodes.

Each node needs to be aware of the request counts from other nodes to enforce the rate limit consistently. Without a distributed solution, each node would only track requests it directly handles, leading to inconsistent rate limiting.

You might think that maintaining request limits in memory is not feasible at a scale that requires multiple nodes. But since each client's state is just a single integer for the token size and a single integer for the timestamp, this doesn't require a large amount of memory (2 * 64 bytes per client). Even if there are 100k concurrent clients, less than 15MB of in-memory state is required.

Understanding GenServers in a Distributed Elixir Environment

Distributed systems pose unique challenges, particularly when it comes to state management. Network partitions, node failures, and latency are just a few issues that can disrupt state consistency across nodes. Ensuring that all nodes have the correct state or can recover it when things go wrong is critical for a system's reliability.

GenServers can be distributed across nodes in a cluster, allowing them to share state and handle requests from different parts of a system. By leveraging the built-in distribution mechanisms of the BEAM VM, GenServers can communicate across nodes with the same ease as within a single node.

In the next section, we will use some strategies to support rate limiting across multiple nodes in the cluster.

Implementing Distributed GenServers Using a Single Global Process

The simplest strategy to support a rate limiter across a whole cluster is to start a single global process that manages the limits. This is simple to do using Erlang's :global name registration.

In the single node setup above, we register the GenServer under the name MyApp.TokenBucketRateLimiter (__MODULE__ contains the name of the current module). This registers the process locally on the node. It is also possible to register it globally using a {:global, name} tuple when calling GenServer.start_link/3. This registers the process under the given name globally across the whole cluster. Note that since we update the process name in start_link, we also have to update it when using any call, cast, and friends. Let's update the code to do this:

defmodule MyApp.TokenBucketRateLimiter do
  def start_link([]) do
    GenServer.start_link(__MODULE__, nil, name: {:global, __MODULE__})
  end

  def allow?(client_id) do
    GenServer.call({:global, __MODULE__}, {:allow?, client_id})
  end
end

This enforces that only a single process with the global name MyApp.TokenBucketRateLimiter can exist across the cluster. However, this is insufficient, as it will prevent all other application nodes from joining our cluster because the supervision tree cannot start all its processes. Let's try it out by starting a few nodes in the cluster.

The first node starts fine:

➜ iex --name node1@127.0.0.1 --cookie asdf -S mix
iex(node1@127.0.0.1)1>

But as soon as we start a second node and connect it to the cluster, the application crashes:

➜ iex --name node2@127.0.0.1 --cookie asdf -S mix
iex(node2@127.0.0.1)1> Node.connect(:"node1@127.0.0.1")
[notice] Application my_app exited: shutdown

We need to make sure that the start_link function of the rate limiter doesn't generate any errors when a process already exists on another node. To do this, we can check and modify the start_link function to return :ignore when the process already exists:

def start_link([]) do
  tuple = {:global, __MODULE__}

  case GenServer.whereis(tuple) do
    nil -> GenServer.start_link(__MODULE__, nil, name: tuple)
    _pid -> :ignore
  end
end

Once that's done, we can start multiple nodes in the cluster, and they will all ping a central rate limiter instance to allow or deny incoming requests. The central rate limiter will be the GenServer process that happens to start first, i.e., the first node in the cluster to come up.

While this works, there are several issues with this approach:

The node is a single point of failure now. If the node goes down, the rate limiter will be unreachable until another node starts it. Even worse, this is not handled automatically, so we might be left with no rate limiter process unless a new node is started.
The state is on a single node. This means that if the rate limiter process is terminated/restarted (either due to a programming error or node failure), the state is lost.
All nodes in the cluster have to make a call to the single node on every API call.

Using Multiple Processes Across the Cluster

In order to provide better fault tolerance when managing state across distributed nodes, we can use a combination of techniques, including:

State partitioning: Dividing the state into partitions so that each GenServer instance handles a subset of the data.
Replication: Replicating state across nodes to provide redundancy and increase fault tolerance.
Conflict resolution: Implementing strategies to handle state conflicts, such as using version vectors or CRDTs (Conflict-free Replicated Data Types).

Let's see how we can use CRDTs to provide super-fast data access alongside eventual consistency and data replication guarantees.

We will use the Elixir library DeltaCrdt to manage shared state across multiple nodes in the cluster.

Note: In addition to DeltaCrdt, other libraries like Horde and Swarm can help you to coordinate processes and state across several nodes in the cluster.

Let's add it to mix.exs to get started:

defmodule MyApp.MixProject do
  defp deps do
    [
      # ...
      {:delta_crdt, "~> 0.6.3"}
    ]
  end
end

DeltaCrdt provides a simple-to-use API for managing a map of data that is guaranteed to be conflict-free (across distributed usage) and replicated across multiple nodes. For conflict resolution, it provides AWLWWMap (that stands for 'Add-Wins Last-Write-Wins Map) so that in the case of a conflict, the last write always wins. The API is simple — you start multiple DeltaCrdt processes (which can be on the same node or multiple nodes) and create a cluster with them using set_neighbours/2:

{:ok, crdt1} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
{:ok, crdt2} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
DeltaCrdt.set_neighbours(crdt1, [crdt2])
DeltaCrdt.read(crdt1)
%{}

Once neighbours are set, adding any data to one CRDT automatically syncs with the second one.

DeltaCrdt.put(crdt1, "foo", "bar")
DeltaCrdt.read(crdt2)
%{"foo" => "bar"}

Starting A One Rate Limiter Per Node

With this out of the way, we can update our implementation of the rate limiter to use DeltaCrdt to sync states. First, we need to make sure all rate limiter processes are discoverable globally, but with unique names. One way to do that is to use the node name when naming the process:

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  def start_link([]), do: GenServer.start_link(__MODULE__, nil, name: global_tuple())

  defp global_tuple(node \\ Node.self()), do: {:global, to_string(node) <> to_string(__MODULE__)}
end

Syncing GenServer State With CRDT

Now that all nodes start their own copy of the rate limiter, we need to sync the state. Let's do that by starting a DeltaCrdt process from the GenServer and using the CRDT to make decisions about the rate limits:

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  @impl true
  def init(nil) do
    {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 100)

    {:ok, %{crdt: crdt}}
  end

  def allow?(client_id), do: GenServer.call(global_tuple(), {:allow?, client_id})

  @impl true
  def handle_call({:allow?, client_id}, _from, state) do
    current_time = System.os_time(:second)

    {reply, updated_bucket} =
      state.crdt
      |> get_or_init_bucket(client_id, current_time)
      |> update_bucket(current_time)
      |> maybe_consume_token(current_time)

    # Update the CRDT with the new bucket state
    DeltaCrdt.put(state.crdt, client_id, updated_bucket)

    {:reply, reply, state}
  end

  defp get_or_init_bucket(crdt, client_id, current_time) do
    case DeltaCrdt.get(crdt, client_id) do
      nil -> default_bucket(current_time)
      bucket -> bucket
    end
  end

  defp default_bucket(time), do: %{tokens: @bucket_size, last_checked: time}
end

The major changes here from the single-node approach are:

Instead of Map.get from the local GenServer state, we use DeltaCrdt.get to get the key from the CRDT instead. This fetches the data from the local copy of the CRDT which is very fast to access and guarantees eventual consistency.
Similar to the above, we use DeltaCrdt.put to update the shared state instead of the local GenServer state when updating the tokens count. This is again a fast operation that updates the local state — but with a guarantee that the changes will be eventually synced with all processes in the cluster.

Clustering the GenServers

While we are almost there, there is one last step left. The CRDTs are currently local and do not sync with other nodes in the GenServer. We need to use the set_neighbours/2 API from DeltaCRDT to set up a sync.

Let's create a new function named update_neighbours inside the GenServer that does this:

defp update_neighbours(crdt) do
  neighbours =
    Node.list()
    |> Enum.map(fn node ->
      node
      |> global_tuple()
      |> GenServer.whereis()
      |> :sys.get_state()
      |> Map.get(:crdt)
    end)

  DeltaCrdt.set_neighbours(crdt, neighbours)
end

Now all we need to do is call this function when the GenServer starts (from init or a handle_continue callback).

The above piece of code takes care of updating CRDT neighbors when a new process starts. But set_neighbours is a one-sided operation (i.e., it sets up puts from the CRDT to its neighbors, but not the other way around). To ensure servers that have already started also update their neighbors, we must call update_neighbours on all cluster processes as soon as any service starts. Let's add a new refresh_peer_neighbours function to do that (which should then be called after the initialization of the GenServer).

defp refresh_peer_neighbours() do
  Node.list()
  |> Enum.each(fn node ->
    node
    |> global_tuple()
    |> GenServer.whereis()
    |> Process.send(pid, :refresh_neighbours, [])
  end)
end

@impl true
def handle_info(:refresh_neighbours, state) do
  update_neighbours(state.crdt)
  {:noreply, state}
end

Handling Cluster Changes

With all this in place, there is still one final piece of the puzzle. What happens when nodes are added or removed from the cluster? To update the CRDT neighborhood on cluster-level changes, we can use :net_kernel.monitor_nodes(true) from Erlang. This monitors the nodes and refreshes the neighbors on any nodeup or nodedown events:

@impl true
def init(nil) do
  {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 100)

  # Monitor for addition of new nodes
  :net_kernel.monitor_nodes(true)

  {:ok, %{crdt: crdt}, {:continue, nil}}
end

@impl true
def handle_info({:nodeup, _node}, state) do
  Process.send_after(self(), :refresh_neighbours, 1_000)
  {:noreply, state}
end

def handle_info({:nodedown, _node}, state) do
  Process.send_after(self(), :refresh_neighbours, 1_000)
  {:noreply, state}
end

This finally brings us to the end of our implementation. There are some edge cases to consider:

Avoiding deadlocks when refreshing neighbours (since we need to fetch the PID of the CRDTs from all nodes).
Handling cases where the RateLimiter does not exist on a node (e.g., when it is being restarted).

Here is the full implementation. Let's take it for a spin:

# Node 1
➜ iex --name node1@127.0.0.1 --cookie asdf -S mix
iex(node1@127.0.0.1)1>

# Node2
➜ iex --name node2@127.0.0.1 --cookie asdf -S mix
iex(node2@127.0.0.1)1> Node.connect(:"node1@127.0.0.1")
true
iex(node2@127.0.0.1)2> MyApp.TokenBucketRateLimiter.allow?("some client id")
true
iex(node2@127.0.0.1)3> MyApp.TokenBucketRateLimiter.crdt() |> DeltaCrdt.get("some client id")
%{tokens: 9, last_checked: 1725358705}

# Back on Node1
iex(node1@127.0.0.1)1> MyApp.TokenBucketRateLimiter.crdt() |> DeltaCrdt.get("some client id")
%{tokens: 9, last_checked: 1725358705}

And that's it for part one of this series!

Wrapping Up

In this first part of our Distributed Phoenix series, we've taken an in-depth look at how GenServers and Delta CRDTs can be leveraged to manage distributed state in a Phoenix application. We've implemented a token bucket rate limiter across multiple nodes, ensuring consistency and fault tolerance even in a distributed setup. This foundational knowledge is crucial for building scalable, resilient systems that can handle high traffic and maintain data integrity across multiple servers.

In part two, we'll dive deeper into advanced techniques for optimizing distributed systems, including strategies for deploying and scaling distributed applications.

Stay tuned for more insights and practical examples that will help you master the art of distributed applications with Phoenix and Elixir.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Find and Fix N+1 Queries Using AppSignal for a Phoenix App in Elixir

Roy Tomeij — 2024-10-08T00:00:00+00:00

N+1 queries are a frequent issue in complex applications built with Elixir and Phoenix. These queries can silently degrade application performance, often going unnoticed until they've compounded into a significant problem.

They can substantially increase web page load times, as each additional query adds overhead to a database, consuming more time and resources. That's why it's crucial to detect and resolve N+1 queries to optimize production systems. It's not just about maintaining a seamless user experience; it's also about ensuring you maintain your Elixir application's scalability and efficiency.

Using AppSignal, you can identify N+1 queries in Phoenix applications, alongside a comprehensive suite of tools to monitor, diagnose, and rectify performance bottlenecks in Elixir projects.

Let's explore how N+1 queries happen, their impact on performance, and some strategies to detect and fix them in your Elixir application.

Understanding And Fixing N+1 Queries in Phoenix for Elixir

N+1 queries occur when an application retrieves a single database record, followed by an additional query for each related record. In Phoenix applications, N+1 queries often emerge in the context of complex relationships and Ecto associations.

Even small mistakes in Ecto queries can lead to cascading N+1 issues, resulting in multiple round trips to a database, each fetching only a small amount of data.

Some common examples include:

Using Repo.all without a proper preload clause (this will not fetch associated records in the same query).
In GraphQL, when using Absinthe with Elixir, N+1 query issues can arise when resolving fields that require data from associated records.

Check out Avoiding N+1 Queries for Absinthe for an overview of this topic.

A Simple Example Using Ecto

For instance, consider a blog application where each post has many comments. A naive Ecto query might retrieve all posts and then run a separate query for each post's comments. This results in one query for the posts (N) and one query for each post's comments (+1), which is highly inefficient.

# Fetching posts without preloading comments
posts = Repo.all(Post)

# This will cause N+1 queries, as for each post, comments are fetched separately
posts |> Enum.each(fn post ->
  comments = Repo.all(from c in Comment, where: c.post_id == ^post.id)
  IO.inspect(comments)
end)

The above might sound like a contrived example — experienced developers are quite unlikely to do this. But it is meant to serve as a simplified scenario of how N+1 queries can creep into systems: any query inside a loop is a suspect. N+1 queries are usually hidden in plain sight, for example, behind a context function like Blog.list_comments(post_id: post.id) inside the loop.

Next, let's discuss a complex scenario.

Complex Situations Leading to N+1 Queries

Imagine a blog application displaying a feed of all recent posts and the count of comments (or likes) for each post.

A sample implementation could look like this:

# Load posts from subscribed authors
posts = Repo.all(Post)

# Load comments count per post
for post <- posts do
  comments_count = Repo.one(from c in Comment, where: c.post_id == ^post.id, select: count(c.id))
  render("post.html", post: post, comments_count: comments_count)
end

Here, the comments_count query causes an N+1 problem because we fetch each individual post's count. One might argue that we can just add comments to the preloads and be done with it. While that solves the N+1 query, it loads too much unnecessary data (we don't need all the comments yet, only the total number of comments) and might actually hurt loading times.

A better solution is to join the tables and use select to load the comments count within the posts query:

posts_with_counts =
  from(p in Post,
    left_join: c in Comment, on: c.post_id == p.id,
    group_by: [p.id],
    select: %{p | comments_count: count(c.id)}
  )
  |> Repo.all()

for post <- posts_with_counts do
  render("post.html", post: post, comments_count: post.comments_count)
end

To avoid and detect N+1 queries, developers must be vigilant when writing Ecto queries and templates. Using preload correctly to fetch all necessary records in a single database query is essential.

Detecting N+1 Queries By Their Impact

Understanding the data access patterns of your application can help you anticipate and prevent N+1 issues before they arise. One way to identify N+1 queries is to observe and analyze common bottlenecks in your app, including:

Increased Load Times: Each query takes time to execute, and when hundreds or thousands of them run sequentially, the cumulative effect leads to noticeable delays in content rendering.
Database Strain: Databases are designed to handle multiple queries efficiently, but an influx of unnecessary queries can strain the system, leading to slower response times for all users.
Server Resource Drain: A server's resources are finite, and handling a multitude of queries consumes more CPU and memory, potentially affecting other operations and leading to resource exhaustion.
Scalability Issues: As a user base grows, inefficient N+1 queries can prevent an application from scaling smoothly, requiring more hardware resources to handle the same workload.
Cost Implications: Increased server load and database usage can lead to higher operational costs, especially in cloud-based environments where resources are metered.

However, it is difficult to spot small differences in performance and then pinpoint the parts of your application that have caused such degradation.

The next sections will show how AppSignal can help you detect and fix N+1 queries, ensuring your application runs smoothly and efficiently.

Detecting N+1 Queries in Phoenix for Elixir Using AppSignal

Before detecting N+1 queries, we need to set up AppSignal in your Phoenix application. You can sign up for a free 30-day trial of AppSignal.

AppSignal provides an Elixir package that integrates seamlessly with Phoenix applications. The AppSignal for Elixir setup process involves adding the Elixir package to your project's dependencies and configuring it with your AppSignal Push API key. Once installed, AppSignal starts monitoring your application's performance, including database query patterns.

Analyzing Elixir Data with AppSignal

AppSignal's instrumentation for Phoenix and Ecto is designed to provide detailed insights into your application's database interactions. It automatically tracks database query times, helping you spot inefficiencies. With AppSignal's instrumentation in place, detecting N+1 queries becomes a matter of analyzing the collected data.

Use Slow Queries to Find N+1 Queries

AppSignal's Slow Queries dashboard breaks down web requests, showing the slowest queries and potential bottlenecks. A series of similar queries within a single web request is a strong indicator of an N+1 query problem.

For example, if you see repeated queries fetching comments for different posts while a blog page is being rendered, you've likely encountered an N+1 issue. AppSignal provides context, including specific database calls, making it easier to pinpoint the exact location in your code where the issue originated.

We can see that the first query had almost 95% impact on all our queries executed through Ecto. In a real-world app, this number might be lower, as there can be hundreds of unique queries. However, any outlier that's highlighted here is a great candidate for analysis.

💡 Note: The "impact" of a query on an application is based on its usage compared to other queries.

Another metric to consider here is the high throughput of 720. Throughput is the total number of queries that were executed in a specified time window.

Let's check out the full details of the query:

SELECT c0."id", c0."body", c0."post_id", c0."user_id", c0."inserted_at", c0."updated_at", c0."post_id" FROM "comments" AS c0 WHERE (c0."post_id" = $1) ORDER BY c0."post_id"

Since the query is targeting a single post, it is quite likely an N+1 query.

Trace N+1 Queries with the Event Timeline

AppSignal's Event Timeline is also particularly useful for finding and tracing N+1 queries. It visualizes a sequence of database calls during a web request, allowing you to identify the hallmark successive database queries that characterize N+1 issues.

Here's an example event timeline for a page that has an N+1 query issue:

We can see that many quick queries are being performed. Hovering over query.ecto also shows the exact query that was performed to help us pinpoint which query is the culprit.

Another good indication of a possible issue is when the ecto time of an event is abnormally high. For example, in this case, we can see that Ecto was the major time-hog:

Addressing Detected N+1 Queries

Once you've identified an N+1 query with AppSignal, the next step is to address it. This typically involves optimizing your Ecto queries to preload associated records.

AppSignal's detailed metrics allow you to compare performance before and after the changes, giving you concrete feedback on the impact of your optimizations.

And that's it!

Wrapping Up

Whether you're a seasoned developer or new to Phoenix, understanding the nuances of N+1 queries and leveraging AppSignal's capabilities will empower you to build faster, more reliable applications.

Remember, the key to maintaining a performant application is not just fixing issues as they arise, but continuously monitoring and improving your codebase with the help of powerful tools like AppSignal.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Complete Guide to Phoenix for Elixir Monitoring with AppSignal

Roy Tomeij — 2024-09-17T00:00:00+00:00

For Phoenix developers, maintaining the health of your applications is critical. AppSignal offers a powerful solution to gain deep insights into your application's performance and stability.

In this introductory guide, we'll walk through the process of setting up AppSignal in your Phoenix app, instrumenting your code for detailed monitoring, handling errors effectively, and utilizing AppSignal's features to maintain and improve your application's performance. The lessons you learn from this article will help you get started using AppSignal to monitor your Phoenix app.

But before proceeding, let's see what you'll need to follow along effectively.

Prerequisites

An AppSignal account — you can sign up for a free 30-day trial.
A Phoenix app to work with. If you don't have one, you can fork this one that we'll use throughout the tutorial.
Some experience working with Elixir/Phoenix applications.

Let's get started by learning how to add AppSignal to an existing Phoenix application.

Adding AppSignal to a Phoenix Application

Adding AppSignal's Phoenix package is very easy — just open up mix.exs and add the package as shown below:

# mix.exs
...
defp deps do
  [
    {:phoenix, "~> 1.7.14"},
    {:phoenix_ecto, "~> 4.5"},
    ...
    {:appsignal_phoenix, "~> 2.0"}
  ]
end
...

After that, run mix deps.get to add the package to the application, then run the command below to install it:

mix appsignal.install <YOUR API KEY HERE>

Tip: You can find your API key in the AppSignal dashboard.

When you run the installer, you'll be prompted for an application name — go ahead and enter an appropriate name:

Validating Push API key: Valid! 🎉
What is your application's name? [counter_live_app]: Counter Live App

Next up, you'll need to choose how you want the AppSignal configuration handled for your app:

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

You can choose whatever suits you, but I prefer option 1, which uses a config file that you can customize to your liking. Once you make your choice, the AppSignal configuration should now be working, and if you visit the dashboard, you can see some default graphs (these might be empty initially as it takes a little bit of time for your app's data to be sent over).

Before moving on, let's take a look at how you can customize the AppSignal config file.

Customizing the AppSignal Config File

A config file tells AppSignal which app and environment to instrument.

The code snippet below shows a minimal configuration:

# config/appsignal.exs

import Config

config :appsignal, :config,
  otp_app: :counter_live_app,
  name: "Counter Live App",
  push_api_key: "API KEY HERE",
  env: Mix.env

As you can see, the most important settings include:

Your application's OTP name.
The name of the app as you want it to appear on AppSignal.
Your AppSignal API key.
The environment you want to instrument for, which could be development, production, or test environments.

The config file can be further customized using a variety of other options.

With the installation and configuration working, let's shift gears to monitor our Phoenix app's performance.

Monitoring the Performance of a Phoenix App

When you add the AppSignal package to your Phoenix application, the package will take care of default HTTP instrumentation needs and send data to a dashboard:

And part of the automatic instrumentation includes performance monitoring:

The default instrumentation will also include important details for slow-performing requests, like what's shown below:

And all of this is possible without any extra work from you.

Next up, let's see how you can customize some of the performance instrumentation. For example, we can add a custom slow function to the Posts index and see the same on the AppSignal dashboard.

First, let's edit the index action in the posts controller as shown below:

# lib/counter_live_app_web/controllers/posts_controller.ex

defmodule CounterLiveAppWeb.PostController do
  ...

  def index(conn, params) do
    really_slow()
    ...
  end
  ...
  def really_slow do
    Appsignal.instrument("really slow index request", fn ->
      :timer.sleep(1000)
    end)
  end
end

And with that, we can visualize the instrumentation on the dashboard:

Tracking Phoenix App Errors Using AppSignal

In the previous section, we saw how adding AppSignal to our Phoenix app provides us with a lot of information about app performance. We also went ahead and added some simple custom instrumentation to simulate a really slow request to the posts index action, and visualized the results on the AppSignal dashboard. Now, let's see how we can view and track Phoenix errors using AppSignal.

Our first example will be a simple call to a non-existent route, as shown below:

Again, as with the default performance metrics, without extra work on our part, AppSignal is able to pick up this error and display it on the dashboard:

If you click on the error, you can see more detailed information, which is really helpful for debugging purposes:

I won't go into more detail on how you can customize exception instrumentation in the scope of this post. If you are interested in going down that rabbit hole, check out the AppSignal for Elixir documentation on exception handling.

Next up, let's set up alerts and notifications.

Setting up Effective Alerts and Notifications

As useful as an application performance monitoring solution like AppSignal is, you wouldn't want to spend all your time staring at dashboards. Instead, it would be better if you received automated alerts for any significant events.

By default, whenever an error or performance event happens, it's recorded as an error or performance incident by AppSignal. Depending on the notification settings you've set up, you'll get an email (the default notification channel) or a message through any of the other available notification channels.

You can customize how AppSignal will notify you of events and errors by setting up any of the five notification defaults available:

Never notify - when set to this, AppSignal will not notify you whenever an error or incident occurs. This setting is useful for non-critical incidents.
Every occurrence - Let's say you'd like to be notified every time an incident happens. This is the setting you would use. You can use this setting for incidents that are critical in nature.
First in deploy - Here, a notification will be sent whenever an incident occurs when you first deploy your app. You will need to set up deploy markers that will tell AppSignal when a deployment occurs. For example, let's say you've set up post-deploy hooks of some kind. In that case, this setting helps you catch errors if these hooks don't work as expected.
First after close - This setting tells AppSignal to send a notification after an incident recurs, immediately after you close a previous similar one. This setting would be useful for incidents that are not entirely due to a bug in your code, (for example, third-party API connection issues and so forth).
Every nth hour or day - Here, you tell AppSignal not to send a notification every nth hour or day. This could be very useful if you are facing an increased bill due to an increase in notification events.

So far, everything we've looked at is quite basic, but your AppSignal setup is capable of a lot more.

Next, let's briefly look at one or two examples of custom monitoring and alert notification setups.

Custom Alert Notification Setup

By default, every application that is being monitored by AppSignal will have an email notifier set up automatically.

Even so, you might want to be notified via other channels, and with AppSignal, these are a breeze to set up. The first step is to click on the "Add integration" button.

For example, if you wanted a Discord notifier, just choose it from the list of integrations and input the required information in the resulting dialog like this:

Even with all the bells and whistles that AppSignal makes available for alerting and notifications, it's important to really think about the kinds of notifications you would like to receive and the ones you'd be safe ignoring.

Some good rules of thumb to guide your decision in this would include:

Consider error thresholds for critical versus non-critical errors. For non-critical errors, you could set very high thresholds. For critical errors, you might set finer thresholds (for example, send an alert if a critical error exceeds 5% and lasts for more than 3 minutes).
Set up alerts and notifications for memory consumption. One of the key resources any app uses is memory. If an app is consuming more-than-normal memory, you want to make sure an alert is sent to you early.
Uptime and availability notifications. This is self-explanatory, but you definitely want to be notified any time your app is unavailable.
It would also make sense to set up alerts for custom events that are key to the functionality of your app, or to the user experience (for example, a bunch of failed payments over a certain amount of time).

With these rules in mind, and considering AppSignal's robust alerting infrastructure, you have all you need to be notified whenever things go wrong. If you need further information, go through AppSignal's documentation on setting up notifications and alerts.

Monitoring of a Phoenix App Using AppSignal: An Example

We'll use AppSignal's function decorators to set up a custom error span that will be fired when we try to request a non-existent post.

First, we modify the posts controller to use AppSignal decorators like this:

# lib/counter_live_app_web/controllers/post_controller.ex

defmodule CounterLiveAppWeb.PostController do
  use CounterLiveAppWeb, :controller
  use Appsignal.Instrumentation.Decorators # add this line

  ...
end

Then we modify the show action with the custom instrumentation as shown below:

# lib/counter_live_app_web/controllers/post_controller.ex

...
  def show(conn, %{"id" => id}) do
    try do
      post = Blog.get_post!(id)
      render(conn, :show, post: post)
    rescue
      e in Ecto.NoResultsError ->
        Appsignal.send_error(e, __STACKTRACE__, fn span ->
          Appsignal.Span.set_name(span, "Custom Post Not Found Error")
          Appsignal.Span.set_sample_data(span, "custom_data", %{
            post_id: id,
            controller: "PostController",
            action: "show"
          })
        end)

        conn
        |> put_status(:not_found)
        |> put_view(CounterLiveAppWeb.ErrorView)
        |> render("404.html")
    end
  end
...

Here, we use AppSignal's send_error/3 to define a custom span with the name "Custom Post Not Found Error", which is called if we make a request to a non-existent post.

As you can see from the screenshots below, the error is captured as expected on AppSignal:

Clicking on the error link gives you access to the error details:

And that's it!

Wrapping Up

In this article, we've looked at how to set up AppSignal to monitor a Phoenix application. We also covered how errors and performance incidences are instrumented and displayed on the dashboard.

Even though this is enough to get you started with monitoring your Phoenix application, AppSignal offers a lot more options for you to explore (such as Ecto-specific monitoring to cover your app's Ecto queries, plug monitoring, and even in-depth custom instrumentation to cover almost any monitoring scenario you might have).

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Scaling Your Phoenix App in Elixir with FLAME

Roy Tomeij — 2024-09-03T00:00:00+00:00

When you build an app, you'll often find that certain tasks do not require user interaction and are better performed in the background. Elixir provides excellent primitives, such as Task.async, to offload these tasks from the main user pipeline. Additionally, libraries like Oban offer more control over background tasks when needed.

There's also FLAME, which the core Phoenix team is developing to offer a scalable solution for offloading intensive tasks to remote machines.

We will compare these methods and see how FLAME stands out.

But first, let's delve into how FLAME works and how it can help you scale your Phoenix application.

Understanding FLAME for Phoenix

FLAME stands for Fleeting Lambda Application for Modular Execution. It is similar to Task.async, allowing you to run any block of code, but with the added benefit of executing it on a separate node. Depending on your configuration, FLAME can automatically manage infrastructure, spawning or scaling down nodes as needed.

To illustrate how FLAME works, let's consider an example where we need to find the SHA-256 hash of a file stored on Amazon S3. Calculating the hash for a large file can prove slow due to file size or network latency. By offloading this task to a background worker with FLAME, we can improve the performance and responsiveness of our main application.

Here’s how you can do it:

FLAME.call(MyApp.BackgroundRunner, fn ->
  ExAws.S3.download_file(bucket, file_path, :memory)
  |> ExAws.stream!()
  |> Enum.reduce(:crypto.hash_init(:sha256), fn chunk, acc ->
    :crypto.hash_update(acc, chunk)
  end)
  |> :crypto.hash_final()
  |> Base.encode16()
end)

This is similar to Task.async and Task.await, with the key difference being that it runs on a separate node. When the checksum is ready, it returns the result to the main node. We'll discuss configuration options and available functions later in the post.

So, how is this different from any background job processing framework (e.g., Oban)?

Built-in Scaling: FLAME has built-in support for scaling. It can automatically spawn new nodes when new tasks come in and scale down to zero (configurable) when no tasks are in the queue.
Minimal Boilerplate: Unlike other frameworks, FLAME does not require extensive boilerplate code. You simply wrap your task inside FLAME.call or FLAME.cast.
Awaiting Results: In most job frameworks (including the free version of Oban), you cannot await the result of background tasks. FLAME, however, supports this feature, allowing you to wait for a task's completion and retrieve the result.

FLAME in Action

In the previous section, we saw how easy it was to run a piece of code on a separate node using FLAME. Let's pull back a little and see how to integrate FLAME inside an existing Phoenix app.

Install FLAME

Add the dependency in mix.exs:

{:flame, "~> 0.1.12"}

Start a FLAME Pool

FLAME.Pool is the main GenServer provided by FLAME that manages the scaling of nodes inside your app. It also schedules tasks and delivers results to each task's respective nodes.

Add FLAME.Pool to your application's supervision tree by updating the application.ex:

defmodule MyApp.Application do
  use Application

  @half_hour_millis 1_800_000

  @impl true
  def start(_type, _args) do
    children =
      [
        # ...
        # The BackgroundRunner pool controlled by FLAME
        {FLAME.Pool, name: MyApp.BackgroundRunner, min: 0, max: 5, max_concurrency: 5, idle_shutdown_after: @half_hour_millis},
      ]
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

The name option defines the pool name to be used in calls to FLAME later in the application. This allows you to define several different pools with different configurations depending on the tasks that you want to achieve.

For example, for nodes that handle hash computation, you might want different concurrency settings than the nodes that handle more complex tasks, like video encoding. Check out all the available options for the pool. The most important ones that you will commonly use are:

:min - The minimum number of nodes to start. You can configure this to zero to support pools that spawn a node only when there is a task, and then scale down when there are no pending tasks. This is especially useful in pre-production environments to save infrastructure costs.
:max - The maximum number of nodes at a time.
:max_concurrency - The maximum number of concurrent executions on a node.
:timeout - Default timeout for functions. This can also be set during individual calls (using FLAME.call with the timeout option).
:idle_shutdown_after - The number of seconds after which an idle node is shut down (only if there are more than the min number of nodes).

Configure FLAME Backend

FLAME Backend is the component responsible for starting up new nodes, connecting them to parent nodes, and running functions. By default, FLAME ships with the FlyBackend. If you run your application on Fly, configuring FLAME is very simple. Just update your config.exs or runtime.exs if you use Elixir releases:

config :flame, :backend, FLAME.FlyBackend

config :flame, FLAME.FlyBackend,
  token: System.fetch_env!("FLY_API_TOKEN"),
  cpu_kind: System.get_env("FLAME_CPU_KIND", "shared"),
  memory_mb: System.get_env("FLAME_MEMORY_MB", "1024") |> String.to_integer()

To access the Fly API for spawning or destroying machines, you need a token from Fly. You can optionally configure the cpu_kind, memory, and cpus for the spawned nodes. Check out the full list of supported options in the FlyBackend docs.

There is also a FLAMEK8sBackend available if you are running on Kubernetes. If you are not using Kubernetes, other platforms have no out-of-the-box support. However, you can refer to the FLAME.FlyBackend code and create a similar backend for your platform.

Running Tasks with FLAME

FLAME provides two main functions for running tasks on other nodes:

FLAME.call/1: This function calls a task on a remote node and waits for the result. It's useful when you need to run tasks in the background but still require a result.

**Example**: A user requests a 10,000,000th Fibonacci number.
You can run the computation on another machine to avoid blocking web server resources while the user waits for the result.

FLAME.cast/1: This function calls a task on a remote node without waiting for a result. It's useful when a result isn't needed immediately and you don't want to block the user pipeline.
```
**Example**: After a user uploads a file, you want to update its checksum in the records.
This can be done in the background without making the user wait.
```

By default, the spawned processes on the remote node are linked to the parent process, preventing orphaned tasks on remote nodes if the parent process is killed or dies. However, for some background tasks (e.g., file checksum generation), it might be useful to continue running a task even if the parent process is terminated.

Both FLAME.call/2 and FLAME.cast/2 support a link option that can be set to false to achieve this.

Deploying FLAME

You don't need anything special to deploy FLAME as long as you have configured a backend with all the required properties. FLAME starts your full application supervision tree on the remote node. Some parts of the application might not be necessary on a worker node. To control this, you can use FLAME.Parent.get/0 to determine if an application is running on the main node or a child node. Update application.ex to add children to your supervision tree only if they are on the main node (i.e., if FLAME.parent.get() is nil):

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    flame_instance? = not is_nil(FLAME.Parent.get())
    children = [
      # ...
      {FLAME.Pool, name: MyApp.BackgroundRunner, min: 0, max: 5, max_concurrency: 5, idle_shutdown_after: @half_hour_millis},
      !flame_instance? && MyAppWeb.Endpoint
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Technical Deep Dive into FLAME

Internally, FLAME leverages Elixir's powerful Node.spawn_monitor/4 function for spawning processes on remote nodes. This, coupled with BEAM's closure behavior, forms the core of FLAME's implementation.

Let's delve into what occurs under the hood when you execute FLAME.cast or FLAME.call:

The request is dispatched to a pool of runners.
If a runner is readily available (based on defined constraints like min, max, etc.), it spawns and monitors the process on the remote node.
- Upon successful execution, the result is returned or discarded depending on the operation used.
- Additionally, it monitors the process according to the timeout configuration and terminates it if necessary.
- If the remote node crashes (e.g., due to memory exhaustion), the parent process is notified or terminated as per the configuration.
If a process cannot be accommodated on an existing remote node, FLAME tries to spawn a new node (using the configured backend) within the specified limits.
- Once the node is spawned, it follows the same steps as mentioned in (2).
- If spawning a new node fails (e.g., because the maximum limit is reached), the task is queued until resources become available.
Regardless of specific user calls, FLAME autonomously manages idle nodes based on the configuration, shutting them down as needed using the configured backend.

Understanding Closures

Closures play a pivotal role in FLAME's operation, and understanding them is essential in BEAM programming. A closure captures its enclosing environment, including variables from the outer scope. When a closure is defined, it retains references to the variables it "closes over", thus preserving the state of its enclosing process. This behavior is crucial for maintaining consistency, even in scenarios where a process crashes and restarts.

However, when transmitting closures across different nodes in BEAM, the code on all nodes must match precisely. The Fly backend addresses this requirement well, as deployments on Fly rely on Docker images of the released code. This ensures uniformity between the parent node and the remote node, facilitating seamless execution of FLAME.

Leveraging Files as Processes

Things get more interesting when we explore the concept of treating files as processes. In Elixir, opening a file spawns a new process. Writing to a file is akin to sending messages to the process handling the file descriptor. This behavior extends to functions like File.stream! and all other file operations. Consequently, we can extend FLAME's capabilities to handle scenarios such as computing checksums of files stored on S3 or even supporting user-uploaded files directly on the web server.

Here's a code sample:

stream = File.stream!(file_path) # this runs on the web server
FLAME.call(MyApp.BackgroundRunner, fn ->
  # this runs on the remote machine
  stream
  |> Enum.reduce(:crypto.hash_init(:sha256), fn chunk, acc ->
    :crypto.hash_update(acc, chunk)
  end)
  |> :crypto.hash_final()
  |> Base.encode16()
end)

In this scenario, the file stream is initiated on the source machine, while the checksum computation is offloaded to the remote worker. This setup capitalizes on Elixir and BEAM's handling of closures and processes. However, it's worth noting that a more efficient approach would involve establishing a shared volume accessible from both nodes. A shared volume minimizes resource usage and enhances performance. This is in contrast to utilizing the stream from the main node, which still consumes resources to transmit data to the remote node.

Comparing FLAME With Other Approaches

Let's now see how FLAME stacks up against some alternative approaches.

Running Background Tasks on the Same Node

Using Task.async and similar functions, your tasks run on the same node as your web server. This approach works well for quick, lightweight tasks. However, during periods of heavy load, these background tasks can compete with your web server for resources, potentially slowing down your application's response time.

Using Oban for Elixir to Get More Control

Oban is a powerful library that allows you to manage background jobs with more granularity. It enables you to:

Define dedicated Oban Workers.
Configure job queues.
Set up a job storage backend.

Oban also supports running these tasks on dedicated nodes, which can help reduce the load on your main web server. However, setting up Oban involves significant boilerplate code. In its free version, it does not support automatic scaling of the physical nodes running the jobs.

Achieving Infinite Scale with Serverless Functions

For truly infinite scalability, you can use external serverless functions like AWS Lambda, Google Cloud Functions, or Azure Functions. These services allow you to run background tasks independently of your main application infrastructure. However, they come with their own set of challenges:

Significant boilerplate and context switching for developers.
Synchronization issues between third-party services and your application code.
Potential latency and cold start problems.

FLAME: A Summary

As we have seen, FLAME provides a robust way to handle complex background tasks at scale within your Phoenix app. It aims to simplify the setup process and reduce boilerplate code, offering an efficient and scalable solution. All of this comes without the need to context switch — you can call FLAME code from within your app.

By using FLAME, you can:

Easily offload tasks from the main user pipeline.
Manage and scale background jobs without extensive configuration.
Improve your application's performance and responsiveness during heavy loads.

Wrapping Up

In this post, we've explored how to scale your Phoenix application using FLAME. We've delved into how FLAME stands out by offloading tasks to remote nodes. It also offers built-in scaling with minimal boilerplate code. We've compared FLAME with other methods, too, like Task.async, Oban, and external serverless functions, highlighting its unique advantages.

Ultimately, FLAME helps you build more robust and scalable applications by leveraging Elixir's strengths in concurrency and distributed computing, all while keeping your development process straightforward and intuitive.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

LiveState for Elixir: An Overview and How to Build Embeddable Web Apps

Roy Tomeij — 2024-08-20T00:00:00+00:00

If you have programmed with Phoenix, you already know what a delight it can be to work with LiveView. LiveView simplifies your development process by moving all state management to the server. This reduces the complexity of coordinating states between the client and server.

LiveState aims to extend a LiveView-like development flow to embeddable web apps. But before we delve deeper into LiveState, let’s first understand what embeddable web apps are.

Embeddable Web Apps

Embeddable web apps are applications designed to be embedded within another website. These apps are typically small and focus on a specific feature. Examples include:

A comments section on a blog post
A contact form on an otherwise static website
A chat bubble or support widget

LiveState brings the ease and efficiency of the LiveView workflow to the development of these embeddable web apps.

What is LiveState for Elixir?

LiveState is a framework that simplifies creating embeddable web applications by maintaining a server's state. This has several benefits:

Centralized State Management: You can avoid the complexities of synchronizing state between the client and server.
Real-time Updates: LiveState enables real-time updates, ensuring that users always see the most current data without needing to refresh the page.
Reduced Client Complexity: Client-side code can remain simple and lightweight.
Enhanced Security: Sensitive data is less exposed to potential security threats on the client side.

As we will see later in this post, building robust and dynamic embeddable web apps using LiveState is straightforward. With minimal effort, you can leverage the power and simplicity of server-side state management.

More Use Cases

In addition to embeddable web apps, LiveState excels in another important area: highly interactive client components. These components can even exist within a traditional LiveView app, where certain parts of the application are managed using LiveState alongside client-side JavaScript frameworks like React, SolidJS, or LitComponents.

Within a traditional LiveView app, some sections might require more interactivity and a richer client-side experience. LiveState can manage these highly interactive parts using client-side frameworks like React, SolidJS, or LitComponents, while still leveraging the server-side state management benefits of LiveView.

Choosing Between LiveView and LiveState

When deciding whether to use LiveView or LiveState, consider the following:

LiveView: Best suited for applications where most of the interactivity and state management can be handled on the server. This simplifies the client-side code and leverages the power of Elixir and Phoenix to manage state and interactivity efficiently.
LiveState: Ideal for scenarios requiring embeddable features or highly interactive client-side components. If parts of your app need to be embedded in other websites or require rich interactivity that benefits from JavaScript frameworks, LiveState offers a flexible and powerful solution.

Example: Creating An Embeddable Phoenix Web App

Let’s see how LiveState works in practice by integrating it into a Phoenix app to create an embeddable web app. We'll build a LiveState component to manage a to-do list. We'll also build a Phoenix channel backend and use LitElement for our front-end component. The front-end component will mostly be plumbing — we'll use LiveState to manage the state and events on the Phoenix backend, similar to LiveView.

Step 1: Add Dependencies

First, add the necessary dependencies to your mix.exs file:

defmodule TodoApp.MixProject do
  use Mix.Project

  defp deps do
    [
      # ...
      {:live_state, "~> 0.8.1"},
      {:cors_plug, "~> 3.0"}
    ]
  end
end

Step 2: Update Your Endpoint

Next, update your Endpoint to set up a socket for the channel:

defmodule TodoAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :todo_app

  socket "/socket", TodoAppWeb.LiveStateSocket
end

Step 3: Create the `Phoenix.Socket`

Then create the Phoenix.Socket:

defmodule TodoAppWeb.LiveStateSocket do
  use Phoenix.Socket

  channel "todos:*", TodoAppWeb.TodosChannel
  @impl true
  def connect(_params, socket), do: {:ok, socket}

  @impl true
  def id(_), do: "todos:*"
end

This is a standard Phoenix Socket. If you have worked with channels before, you may already be familiar with how this works. For new users, the channel macro defines a channel matching a given topic.

Note that we use simple implementations for the connect and id callbacks that allow all connections. This keeps the guide straightforward, but in a real-world app, you would likely modify them to accept only authenticated users.

Step 4: Define the Channel

Now, let's define the channel. This will be the core of our backend, which will be responsible for maintaining the server-side state and handling events dispatched from the client side. It functions similarly to a Phoenix.LiveView in a traditional setup.

defmodule TodoAppWeb.TodosChannel do
  use LiveState.Channel, web_module: TodoAppWeb

  alias LiveState.Event

  @impl true
  def init(_channel, _payload, _socket) do
    {:ok, %{todos: list_todos()}}
  end

  defp list_todos() do
    Todos.list_todos()
    |> Enum.map(&todo/1)
  end

  defp todo(%Todos.Todo{} = todo), do: Map.take(todo, [:id, :title, :completed])
end

This is the most basic setup for the channel. Let's break it down.

A new channel is initialized as soon as a client connects to the WebSocket and subscribes to the todos:* topic. In the init callback, we add all existing todos to the current state, making this state available to the client side.

We will expand on this module later. First, let's look at the final part of our setup: the client component.

Build the Front-End Component with `LitElement`

We'll use LitElement, so navigate to the assets directory and install lit and phx-live-state.

# assets
$ npm install lit phx-live-state --save

If you are new to LitElement and custom HTML elements, now is a good time to review the basic concepts.

Next, create a new custom element to render the todos inside assets/js:

// assets/js/modules/todos/TodoListElement.ts
import { html, LitElement } from "lit";
import { customElement, property, state } from "lit/decorators.js";
import LiveState, { connectElement } from "phx-live-state";

type Todo = {
  id: number;
  title: string;
  completed: boolean;
};

@customElement("todo-list")
export class TodoListElement extends LitElement {
  @property({ type: String })
  url: string = "";

  @state()
  todos: Array<Todo> = [];

  connectedCallback() {
    super.connectedCallback();
    const liveState = new LiveState({
      url: this.url,
      topic: `todos:${window.location.href}`,
    });

    connectElement(liveState, this, {
      properties: ["todos"],
    });
  }

  render() {
    return html`
      <ul style="list-style-type: none">
        ${this.todos?.map(
          (todo) => html`
            <li>
              <input
                type="checkbox"
                id=${`todo-${todo.id}`}
                ?checked=${todo.completed}
              />
              <label for=${`todo-${todo.id}`}>${todo.title}</label>
            </li>
          `,
        )}
      </ul>
    `;
  }
}

Let's break this down.

TodoListElement Component

First, we define a new LitElement component named TodoListElement using the @customElement decorator. This component is responsible for fetching and displaying a list of todos.

@property: The url property specifies the endpoint for fetching todos.

@state: The todos array holds the list of to-dos fetched from the server.

LiveState Integration

In the connectedCallback method, we set up LiveState to synchronize our component's state with the server.

LiveState is configured with the specified URL and a topic derived from the current page's URL. Using the page's URL is just an example; it can be utilized on the server side to separate to-dos based on the page the component is embedded on.

The connectElement function links the todos state in our component with the LiveState instance, ensuring real-time updates.

There are additional options available with connectElement that we will discuss later.

Rendering the To-do List

The render method generates the HTML structure for our to-do list, displaying each to-do item with a checkbox and label.

Now you can include this component inside your app's JavaScript file (assets/js/app.js):

import "./modules/todos/TodoListElement";

This will make the component available for use. Since we configured it as a custom HTML element, it's simple to use on a page. You can just use the todo-list element and pass a URL (configured as a property) to it.

<todo-list url="ws://127.0.0.1:4000/socket"></todo-list>

If you already have some to-dos, you should now start seeing them on the page.

Adding New To-dos

Next, let's add a way to create new to-dos.

First, we'll update the client to create new to-dos. Modify the connectedCallback to send an add_todo event and receive the todo_created event from the server.

connectedCallback() {
  // ...
  connectElement(liveState, this, {
    properties: ['todos'],
    events: {
      send: ['add_todo'],
      receive: ['todo_created']
    }
  });
}

Then update the render method to include a form that will send the add_todo event to the server:

render() {
  return html`
    <ul style="list-style-type: none">
      ${this.todos?.map(todo => html`
        <li>
          <input type="checkbox" id=${`todo-${todo.id}`} ?checked=${todo.completed}>
          <label for=${`todo-${todo.id}`}>${todo.title}</label>
        </li>
      `)}
    </ul>
    <div>
      <form @submit=${this.addTodo}>
        <div>
          <label for="todo">Todo</label>
          <input id="todo" name="title" required>
        </div>
        <button>Add Todo</button>
      </form>
    </div>
  `;
}

Notice that we referenced the addTodo method in the form's @submit callback. Let's add that method as well:

@query('input[name="title"]')
titleInput: HTMLInputElement | undefined;

addTodo(e: Event) {
  this.dispatchEvent(new CustomEvent('add_todo', {
    detail: {
      title: this.titleInput?.value
    }
  }));
  e.preventDefault();
}

The implementation is straightforward. We need to get the value of the title input and dispatch an event. The event name should match what we configured to send inside the connectElement call.

We used another directive, @query, from LitElement to query the title input. This is a convenient method to find the element matching a selector.

Server-Side Handling

On the server side, we need to handle the add_todo event to create a new to-do and update the state. Let's go back to the TodosChannel and add this functionality.

defmodule TodoAppWeb.TodosChannel do
  alias LiveState.Event

  @impl true
  def handle_event("add_todo", todo_params, %{todos: todos}) do
    case Todos.create_todo(todo_params) do
      {:ok, t} ->
        data = todo(t)
        {:reply, [%Event{name: "todo_created", detail: data}], %{todos: todos ++ [data]}}
    end
  end
end

LiveState calls the handle_event callback when events are dispatched from the client. The event's detail is passed from the client as the second argument to the handle_event callback, while the third argument is the existing channel state, which contains the current to-dos.

In our implementation, we simply create a new to-do and send a :reply back to the client. If no reply is needed for certain events, you can return a {:noreply, state} tuple instead.

In the reply, we send a new LiveState.Event and update the state to include the newly created to-do.

When you run the example again, you will notice that you can now add new to-dos using the form, and they will appear on the list.

You can add more events to toggle to-dos, but we won't go into those in this post. If you're interested, check out the full code sample.

Bundle Embed Script

If you have been following closely, you'll notice that we integrated a component directly into a Phoenix app. In a typical Phoenix app, there's usually no need to do this if the component is simple enough, as you can create views using LiveView.

The purpose of this example was to create a script that someone can embed on any website to get access to the <todo-list> element. Creating the embeddable script from here is straightforward. There are various ways to do this depending on your app's configuration. We'll walk through it using esbuild, which Phoenix configures by default for new apps.

First, create a new entry point for our embeddable component that only imports the TodoListElement:

// app/js/modules/todos/index.ts
import "./TodoListElement";

Next, configure esbuild to generate a new JavaScript file from this entry point. Update config/config.exs:

config :esbuild,
  version: "0.17.11",
  # this is the default entrypoint that phoenix generates
  todo_app: [
    args:
      ~w(js/app.js --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),
    cd: Path.expand("../assets", __DIR__),
    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}
  ],
  # add a new config that bundles only the modules/todos/index.ts file
  todos: [
    args:
      ~w(js/modules/todos/index.ts --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),
    cd: Path.expand("../assets", __DIR__),
    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}
  ]

If you want the embeddable JavaScript to be generated during development, update config/dev.exs to add a new watcher:

config :todo_app, TodoAppWeb.Endpoint,
  # other endpoint config
  watchers: [
    # ...
    # the default watched generated by phoenix
    esbuild: {Esbuild, :install_and_run, [:todo_app, ~w(--sourcemap=inline --watch)]},
    # a new watcher that just watches the todo element (matches the `todos` from config.exs)
    esbuild_todos: {Esbuild, :install_and_run, [:todos, ~w(--sourcemap=inline --watch)]}
  ]

Restart your server, and you should see a new JavaScript file generated inside priv/static/assets. If you embed this component on an external website, include the new JS file and add the <todo-element> as a regular HTML element.

Integrating With JS Frameworks

We've seen an example of integrating LiveState with LitElement, but this is not a requirement. You can also use vanilla JavaScript, adding the phx-live-state package to directly dispatch and listen to events on the LiveState instance without using connectElement.

To use React, the author of LiveState provides a use-live-state hook that simplifies the integration process.

Using this hook, you can efficiently manage the state and events of your to-do list within a React component. This makes it easy to synchronize the client-side state with the server in real time.

export function TodoList() {
  const [state, pushEvent] = useLiveState(liveState, {});
  return (
    <ul style="list-style-type: none">
      ${state.todos?.map((todo) => (
        <li key={todo.id}>
          <input type="checkbox" id={`todo-${todo.id}`} checked={todo.completed}>
          <label for={`todo-${todo.id}`}>{todo.title}</label>
        </li>
      ))}
    </ul>
  );
};

And that's it for now!

Wrapping Up

LiveState brings the simplicity and power of LiveView to embeddable web apps and highly interactive client components.

By maintaining server state, LiveState ensures centralized state management, real-time updates, reduced client complexity, and enhanced security.

It is especially useful if parts of your Elixir application need to be embedded in other websites or require rich interactivity with JavaScript frameworks.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Custom Instrumentation for a Phoenix App in Elixir with AppSignal

Roy Tomeij — 2024-08-06T00:00:00+00:00

In the first part of this series, we saw that even if you just use AppSignal’s default application monitoring, you can get a lot of information about how your Phoenix application is running.

Even so, there are many ways in which a Phoenix application may exhibit performance issues, such as slow database queries, poorly engineered LiveView components, views that are too heavy, or non-optimized assets.

To get a grasp on such issues and peer even more closely into your app’s internals, you’ll need something that packs more punch than the default dashboards: custom instrumentation.

In this article, we'll go through a step-by-step process to build custom instrumentation using AppSignal for a Phoenix application.

Pre-requisites

Before we get into custom instrumentation, here's what you need to follow along:

A Phoenix LiveView app with the AppSignal package installed. If you have an app ready but haven't installed the AppSignal package, just follow this guide. Or you can clone the Phoenix app we'll be using throughout the tutorial.
Basic experience working with Elixir and the Phoenix framework.
An AppSignal account. Sign up for a free trial if you don't have one already.

What Is Custom Instrumentation, and Why Do We Need It?

Custom instrumentation means adding additional telemetry emission and monitoring functionality to your Phoenix app beyond what is provided by the default instrumentation. Specifically, it involves the use of functions and macros (provided through the AppSignal package) in specific places in your codebase that you want to investigate further. You can then view the collected data on a custom AppSignal dashboard.

If you have a complex or mission-critical Phoenix app, it might not be possible to identify all its potential performance issues using the default instrumentation. It's important to have deeper insights into what's going on inside your app.

With custom instrumentation, you can easily structure exactly what you need to identify and visualize these custom data collections within the AppSignal dashboard.

Getting Started with Custom Instrumentation in AppSignal

You can implement custom instrumentation using AppSignal in two ways: through function decorators or instrumentation helpers.

Function Decorators

A function decorator is a higher-order function that you wrap around a function from which you want to collect data. They give you more granular control than instrumentation helpers, but compared to the latter, they are not as flexible.

Instrumentation Helpers

AppSignal also provides instrumentation helper functions, which you can use to instrument specific parts of your code manually. These helper functions can start and stop custom measurements, help you track custom events, and add custom metadata to metrics reported via AppSignal dashboards. This approach gives you more flexibility in instrumenting your code but requires more manual intervention.

In the following sections, we'll use what we've learned to implement custom instrumentation for a Phoenix application.

How to Use an Instrumentation Helper

To implement the first custom instrumentation, we'll consider a controller action that calls a potentially slow function in a simple Phoenix app featuring games and reviews.

As you can see below, the games context has a method for fetching an individual game and preloading any reviews:

# lib/game_reviews_app/games.ex

...
def get_game_with_reviews(id), do: Repo.get(Game, id) |> Repo.preload([:reviews])
...

And the subsequent action using this method in the game controller is:

# lib/game_reviews_app_web/controllers/game_controller.ex

...
def show(conn, %{"id" => id}) do
  game=Games.get_game_with_reviews(id)
  render(conn, :show, game: game)
end
...

Now let's modify this controller action with a custom instrumentation helper to check how many times it is called and its response times. We can do this using AppSignal's instrument/2 function, which takes two parameters:

The function name
The function being instrumented

Note: If you use instrument/3 instead, it's possible to add an event group name as an optional parameter. Usually, whenever you use instrument/2, measurements will be collected and categorized under the "other" event group within the event groups section. But if you wanted another more descriptive name, instrument/3 allows you to pass in an additional parameter for the event group name.

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  use GameReviewsAppWeb, :controller

  ...

  def show(conn, %{"id"=>id}) do
    game = Games.get_game_with_reviews(id)
    am_i_slow() # add this line
    render(conn, :show, game: game)
  end
  ...
end

Let's define the am_i_slow function as a private module:

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  use GameReviewsAppWeb, :controller

  ...
  defp am_i_slow do
    Appsignal.instrument("Check if am slow", fn->
      :timer.sleep(1000)
    end)
  end
end

Specifically, we're defining a custom trace span with an event sample called "Check if am slow". This will show up in our dashboard under the other (background) event namespace, as you can see in the screenshots below:

But what if you wanted a different event group name from "other" or "background" for the event namespace? Just use the instrument/3 function:

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  use GameReviewsAppWeb, :controller

  ...
  defp am_i_slow do
    Appsignal.instrument("Really Slow Queries","First Slow Query", fn->
      :timer.sleep(1000)
    end)
  end
end

Here, we give the span a category name of "Really Slow Queries" and a span name of "First Slow Query", for a sample view like this:

Here's another example of using function helpers:

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  use GameReviewsAppWeb, :controller

  ...
  def index(conn,_params) do
    games = Appsignal.instrument("Fetch games", fn->
    Games.list_games()
  end)

  render(conn, :index, games: games)
  end
end

Using the instrument/2 function, we define a span called "Fetch games" which results in this event trace:

With that, you can easily visualize the function's response time and throughput.

There are more options available to customize AppSignal's instrumentation helpers than what I've shown here. I highly encourage you to check out the possibilities.

Next, let's see how you can use function decorators.

Using Function Decorators

As you've probably noticed when using instrumentation helpers, you end up modifying existing functions with the helper code you add. If you don't want to do this, you can use function decorators instead.

Let's continue working with the game controller and instrument the index method using a decorator. First, we will add the decorator module Appsignal.Instrumentation.Decorators:

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  ...
  use Appsignal.Instrumentation.Decorators
  ...
end

You now have access to the decorator's functions. Let's decorate the index method as shown below:

# lib/game_reviews_app_web/controllers/game_controller.ex

defmodule GameReviewsAppWeb.GameController do
  ...
  use Appsignal.Instrumentation.Decorators
     
  def show(conn, %{"id"=>id}) do
    am_i_slow()
    game = Games.get_game_with_reviews(id)
    render(conn,:show,game:game)
  end

  # add the decorator function
  @decorate transaction_event()
  defp am_i_slow do
    :timer.sleep(1000)
  end

end

This will create a transaction event, which you can visualize in your Events dashboard, as shown below:

You get all the information you need, namely:

a. The resource where the function decorator was called
b. A sample breakdown showing how long Ecto queries took, how long the templates took to load, and so forth.
c. An event timeline with a breakdown of the transaction time of everything involved in that function.

Finally, let's take a look at instrumenting Phoenix channels.

Instrumenting Phoenix LiveViews and Channels

In the example below, we have the welcome live view as shown:

# game_reviews_app_web/live/welcome_live.ex

defmodule GameReviewsAppWeb.WelcomeLive do
  use GameReviewsAppWeb, :live_view

  def mount(_params,_session,socket) do
    {:ok,assign(socket, current_time: DateTime.utc_now())}
  end

  def render(assigns) do
    ~H"""
      <div class="container">
        <h1>Welcome to my LiveView App</h1>
        <p>Current time: <%= @current_time %></p>
      </div>
    """
  end
end

Let's use an instrumentation helper to see how this live view performs:

# game_reviews_app_web/live/welcome_live.ex

defmodule GameReviewsAppWeb.WelcomeLive do
  ...

  import Appsignal.Phoenix.LiveView,only:[instrument: 4]

  def mount(_params,_session,socket) do
    instrument(__MODULE__,"Liveview instrumentation", socket, fn->
      :timer.send_interval(1000,self(),:tick)
      {
        :ok,
        assign(socket,current_time:DateTime.utc_now())
      }
    end)
  end

  ...
end

And as you can see, the transaction times and throughput are now available for inspection on our dashboard:

It's also worth noting that the AppSignal for Elixir package enables you to instrument Phoenix channels using a custom function decorator:

defmodule GameReviewsAppWeb.VideoChannel do
  use GameReviewsAppWeb, :channel

  # first add the instrumentation decorators module
  use Appsignal.Instrumentation.Decorators

  # then add the decorator function
  @decorate channel_action()
  def join("videos:" <> video_id, _params, socket) do
    {:ok,assign(socket, :video_id, String.to_integer(video_id))}
  end
end

And that's it!

Wrapping Up

In part one of this series, we looked at how to set up AppSignal for an Elixir app and AppSignal's error tracking functionality.

In this article, we've seen how easy it is to use AppSignal's Elixir package to implement custom instrumentation for a Phoenix application. We've also learned how to use instrumentation helpers and function decorators.

With this information, you can now easily decide when to use a decorator versus an instrumentation helper in your next Phoenix app.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Enhancing Your Elixir Codebase with Gleam

Roy Tomeij — 2024-07-23T00:00:00+00:00

Do you write Elixir but sometimes miss the benefits of type safety we have in other languages? If the answer is "yes", you may want to take a look at Gleam. It is a relatively young language that runs on top of the BEAM platform, which means it can be added as an enhancement to an Elixir codebase without you having to rewrite everything.

In this article, we will add Gleam code to an Elixir project.

Why Use Gleam for Elixir?

First, let's ask ourselves a question: why would we use Gleam? The Elixir ecosystem is mature at this point, with a multitude of great solutions for everyday problems. Ecto and Phoenix make a really powerful combination for writing web applications. Elixir itself is a dynamically typed language, which is great for some applications but opens the door to a whole class of errors.

Imagine you're working on a critical payment processing system in Elixir and need to ensure your core logic's absolute reliability and correctness. Despite Elixir's strengths, its dynamic typing sometimes leaves room for subtle bugs. Enter Gleam. It's a statically typed language for the BEAM platform that promises to enhance your system's robustness. Where Elixir falls a bit short, Gleam can shine. Let's explore how to integrate Gleam with an Elixir project, so we can have the best from both languages.

We want our core business logic to be written in Gleam while the surrounding code is in Elixir (somewhat akin to the idea of a "functional core, imperative shell", although in the functional world, it's rather a "pure core, stateful shell"). We will take this idea very far, establishing a physical boundary between the two worlds.

Let's see how to get there.

About The Project

To witness the power of Gleam in action, we will implement a feature in a made-up application to manage students at a university. We will take care of their enrollment in courses. Our business rules will look as follows:

A student can be enrolled in a course.
Every course has a limited number of seats and a finite-length waitlist. If a student tries to enroll but all the seats are already taken, they are put on the waitlist (if there are spots there; if not — the enrollment is rejected).
If someone with a reserved seat cancels their enrollment, the first person on the waitlist takes their place.
Some courses have age limits: only students older than a certain age can enroll.

We will start the implementation by creating a fairly standard Phoenix and Ecto application:

mix phx.new student_roll
mix ecto.create
mix phx.gen.live Enrollment Student students name:string date_of_birth:date
mix phx.gen.live Enrollment Course courses name:string max_students:integer waitlist_size:integer min_age:integer
mix phx.gen.schema Enrollment.Enrollment enrollments student_id:integer course_id:integer waitlisted_at:datetime
mix ecto.migrate

With this basic structure in place, we can add Gleam into the mix using the mix_gleam library. We will follow the steps from the project README. First, install mix_gleam:

mix archive.install hex mix_gleam

Now prepare the project to use Gleam by adding some entries to mix.exs's project definition:

@app_name :student_roll

def project do
  [
    archives: [mix_gleam: "~> 0.6"],
    app: @app_name,
    compilers: [:gleam | Mix.compilers()],
    aliases: [
      "deps.get": ["deps.get", "gleam.deps.get"]
    ],
    erlc_paths: [
      "build/dev/erlang/#{@app_name}/_gleam_artefacts"
    ],
    erlc_include_path: "build/dev/erlang/#{@app_name}/include",
    prune_code_paths: false,
    # rest of the function
  ]
end

Note that this assumes you are changing an existing mix.exs file generated by phx.new. These are required steps to make the Gleam code work with Elixir, as outlined in the documentation. You don't need to understand every single change here (I, for instance, don't). The important thing is erlc_path which tells the compiler where to find compiled Gleam files, so we can use them in our project.

Next, also in mix.exs, we need to add the Gleam standard library and testing framework to our dependencies:

defp deps do
  [
    # others
    {:gleam_stdlib, "> 1.0"},
    {:gleeunit, "~> 1.0", only: [:dev, :test], runtime: false}
  ]
end

Finally, fetch dependencies and create the src directory where the Gleam code will live:

mix deps.get
mkdir src

Let's Write Some Gleam

With all the setup in place, we can now start writing our business logic in Gleam! In the first iteration, we will skip the waitlist functionality, but we will still create a system that checks if enrollment is possible. Let's start with a src/enrollment.gleam file. src is the directory at the top level of the project, where mix_gleam expects you to keep your Gleam code.

We will put the following code in the file:

pub type Student {
  Student(id: Int, age: Int)
}

pub type Course {
  Course(id: Int, min_age: Int, seats: Int, seats_used: Int)
}

pub type RejectionReason {
  NoSeats
  AgeRequirementNotMet
}

pub type EnrollmentDecision {
  Enrolled
  Rejected(reason: RejectionReason)
}

pub fn enroll(student: Student, course: Course) -> EnrollmentDecision {
  case student.age >= course.min_age {
    False -> Rejected(AgeRequirementNotMet)
    True ->
      case course.seats > course.seats_used {
        False -> Rejected(NoSeats)
        True -> Enrolled
      }
  }
}

There are quite a few things to unpack here, so let's take a quick crash course on Gleam.

What's Happening Here?

In the first part, we define a bunch of types. Types are the heart of typed languages. Here we need a Student, a Course, an EnrollmentDecision, and a RejectionReason.

Taking a Student declaration, we define not only a type itself but also its constructor, a Student with two arguments: id and age. You can note that we don't include the name here, even though we defined it before in the Elixir schema definition. The name is just side-data. We don't use it for anything related to business logic (unless you have a requirement, such as that only people with names starting with E can take a course).

A type like RejectionReason defines two constructors, which we can basically read as no seats left or the age requirement not being met.

At the end of the code block, we define an actual enroll function. It takes a student and a course, and using a case statement, makes some decisions about the pending enrollment command. case is the only flow control structure in Gleam (there is no if, for example). In this example, we first check the age requirement, and if it's met, we check if seats are left.

With that code ready, we can now write a failing unit test. Let's add a student_roll_test.gleam file in the test directory (this is a naming convention that mix_gleam expects). In that file, add the following:

import enrollment.{Course, Enrolled, Student}
import gleeunit

pub fn main() {
  gleeunit.main()
}

pub fn enrolled_test() {
  let course = Course(id: 1, min_age: 21, seats: 10, seats_used: 9)
  let student = Student(id: 10, age: 20)
  let assert Enrolled = enrollment.enroll(student, course)
}

There's a bit of boilerplate here, but in essence, we create a test function that builds a course with id = 1, min_age = 21, seats_limit = 10 and seats_taken = 9.

Then, we create a student who is 20 years old (therefore too young to enroll). Finally, we try to enroll them. On running mix gleam.test, you get the error:

Running student_roll_test.main
F
Failures:

  1) enrollment_test.enrolled_test: module 'enrollment_test'
     #{function => <<"enrolled_test">>,line => 12,
       message => <<"Assertion pattern match failed">>,
       module => <<"enrollment_test">>,
       value => {rejected,age_requirement_not_met},
       gleam_error => let_assert}
     location: enrollment_test.enrolled_test:12
     stacktrace:
       enrollment_test.enrolled_test
     output:

Finished in 0.011 seconds
1 tests, 1 failures

This is good, but not great. Let's improve the output by using gleeunit/should, a library that makes the unit testing experience better in Gleam:

import enrollment.{Course, Enrolled, Student}
import gleeunit
import gleeunit/should

pub fn main() {
  gleeunit.main()
}

pub fn enrolled_test() {
  let course = Course(id: 1, min_age: 21, seats: 10, seats_used: 9)
  let student = Student(id: 10, age: 20)
  enrollment.enroll(student, course)
  |> should.equal(Enrolled)
}

Now, the output is much more readable:

Running student_roll_test.main
F
Failures:

  1) student_roll_test.enrolled_test: module 'student_roll_test'
     Values were not equal
     expected: Enrolled
          got: Rejected(AgeRequirementNotMet)
     output:

Finished in 0.011 seconds
1 tests, 1 failures

Now, let's fix the test by calling it using a student who is old enough, and we will have a pass:

Running student_roll_test.main
.
Finished in 0.012 seconds
1 tests, 0 failures

Now that we have the first test figured out, we should write some more tests to cover interesting branches in the code. But one important question still lingers: How do I call this from my Phoenix project?

Calling Gleam from Elixir

Let's start working on the Phoenix side of our project. Go to lib/student_roll/enrollment.ex and define two functions related to the enrollment process:

def enrolled?(course_id, student_id) do
  from(e in Enrollment, where: e.student_id == ^student_id
    and e.course_id == ^course_id and is_nil(e.waitlisted_at))
  |> Repo.exists?()
end

def enroll(course_id, student_id) do
  student = get_student!(student_id)
  course = get_course!(course_id)

  # some logic should go here

  %Enrollment{}
  |> Enrollment.changeset(%{student_id: student.id, course_id: course.id})
  |> Repo.insert()
end

We will write a test checking the enrollment logic:

describe "enrollment" do
  import StudentRoll.EnrollmentFixtures

  test "enroll a student" do
    birthday = DateTime.utc_now() |> DateTime.add(-18 * 365, :day)
    student = student_fixture(%{date_of_birth: birthday})
    course = course_fixture(%{min_age: 22})

    assert {:ok, _} = Enrollment.enroll(course.id, student.id)
    assert Enrollment.enrolled?(course.id, student.id)
  end
end

And it passes! But it should not. We created a student who is roughly 18 years old, but the course requires that the students be at least 22. This is because we did not plug our Gleam-written business logic into the mix. Let's fix this now.

Calling compiled Gleam modules looks to Elixir the same way as Erlang modules. You invoke modules by prepending : to the module name. In our case, this will be :enrollment. We will also have to pass something that Gleam can interpret as its Student and Course types. This is the boilerplate I mentioned earlier.

def enroll(course_id, student_id) do
  student = get_student!(student_id)
  course = get_course!(course_id)

  student_age =
    DateTime.utc_now() |> DateTime.to_date() |> Date.diff(student.date_of_birth) |> div(365)

  seats_taken =
    Repo.aggregate(
      from(e in Enrollment, where: e.course_id == ^course_id and is_nil(e.waitlisted_at)),
      :count
    )

  case :enrollment.enroll(
        {:student, student.id, student_age},
        {:course, course.id, course.min_age, course.max_students, seats_taken}
      ) do
    :enrolled ->
      %Enrollment{}
      |> Enrollment.changeset(%{student_id: student.id, course_id: course.id})
      |> Repo.insert()

    {:rejected, reason} ->
      {:error, reason}
  end
end

Our enroll function is slightly inflated, so let's walk through it.

First, we need some more data. We calculate the student age in Elixir here and also fetch the number of already enrolled students. Second, we build a structure that Gleam will interpret as its type. These are just tuples, where the first element is the name of a type, followed by several fields required by the constructor.

# Gleam: Student(id: Int, age: Int)
{:student, student.id, student_age}

# Gleam: Course(id: Int, min_age: Int, seats: Int, seats_used: Int)
{:course, course.id, course.min_age, course.max_students, seats_taken}

Finally, we call :enrollment.enroll with these tuples. If we run the test now, we will get:

1) test enrollment enroll a student (StudentRoll.EnrollmentTest)
  test/student_roll/enrollment_test.exs:131
  match (=) failed
  code:  assert {:ok, _} = Enrollment.enroll(course.id, student.id)
  left:  {:ok, _}
  right: {:error, {:rejected, :age_requirement_not_met}}
  stacktrace:
    test/student_roll/enrollment_test.exs:136: (test)

This is exactly what we wanted to have! The test does not pass because the student is too young.

pub type RejectionReason {
  NoSeats
  AgeRequirementNotMet
}

pub type EnrollmentDecision {
  Enrolled
  Rejected(reason: RejectionReason)
}


# in Elixir:
# :enrolled | {:rejected, :age_requirement_not_met} | {:rejected | :no_seats}

We've made the first step. We called the Gleam code from Elixir, got the results back, and interpreted them back into Elixir idioms (a result tuple of {:ok, term()} | {:error, term()}).

We should now write any remaining tests we want to run in Elixir. Remember that at this point, the enrollment process is thoroughly unit tested in Gleam by Gleeunit, so perhaps we don't need to duplicate all the cases — only the ones that have consequences in Elixir (but this is up to you and your testing strategy).

After this, we can refactor the enroll function to look nicer and more reader-friendly.

defp get_gleam_student!(id) do
  student = get_student!(id)
  student_age =
    DateTime.utc_now() |> DateTime.to_date() |> Date.diff(student.date_of_birth) |> div(365)
  {:student, id, student_age}
end

defp get_gleam_course!(id) do
  course = get_course!(id)
  seats_taken =
    Repo.aggregate(
      from(e in Enrollment, where: e.course_id == ^id and is_nil(e.waitlisted_at)),
      :count
    )
  {:course, course.id, course.min_age, course.max_students, seats_taken}
end

def enroll(course_id, student_id) do
  course = get_gleam_course!(course_id)
  student = get_gleam_student!(student_id)

  case :enrollment.enroll(student, course) do
    :enrolled ->
      %Enrollment{}
      |> Enrollment.changeset(%{student_id: student_id, course_id: course_id})
      |> Repo.insert()

    {:rejected, reason} ->
      {:error, reason}
  end
end

This looks better. You could even hide get_gleam_student!/1 and get_gleam_course!/1 in a module called, for example, GleamTypes. This way, other contexts that potentially call these structures in Gleam will have them readily available.

But that's not all. We still have some work to do.

Implementing the Waitlist

If you recall, our initial requirements also included a waitlist.

This is missing in the implementation above; now is the time to fix it. This will allow you to see how changing the code in both Gleam and Elixir works. I strongly recommend you change the logic in Gleam first, unit-test it, and only then, write the Elixir proxy code (although if you fancy some strict TDD, you can start with some red Elixir tests too).

But before diving into the code, let's ask ourselves one design question: How should we represent the waitlist? Should it be a number (waitlist_size) or maybe a list of students? This is the classic question in functional design and probably deserves a separate article to address it fully.

In our project, for already-signed-in students, we just pass a number of them, not a list of them. This time, we will model the waitlist as an actual list to make things more interesting.

First, we need to change our domain model by extending the Course type:

pub type Course {
  Course(
    id: Int,
    min_age: Int,
    seats: Int,
    seats_used: Int,
    waitlist: List(Student),
    max_waitlist_size: Int,
  )
}

And then the list of enrollment decisions:

pub type EnrollmentDecision {
  Enrolled
  Rejected(reason: RejectionReason)
  Waitlisted
}

After that, we have to adjust our existing Gleam tests because now they fail with the following message:

error: Incorrect arity
   ┌─ /home/user/dev/student_roll/_build/dev/lib/student_roll/test/student_roll_test.gleam:10:16
   │
10 │   let course = Course(id: 1, min_age: 21, seats: 10, seats_used: 9)
   │                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Expected 6 arguments, got 4

This call accepts these additional labelled arguments:

  - max_waitlist_size
  - waitlist

And we also have to modify the Elixir glue code that calls Gleam:

defp get_gleam_student!(id) do
  get_student!(id)
  |> to_gleam_student()
end

defp to_gleam_student(student) do
  student_age =
    DateTime.utc_now() |> DateTime.to_date() |> Date.diff(student.date_of_birth) |> div(365)

  {:student, student.id, student_age}
end

defp get_gleam_course!(id) do
  course = get_course!(id)

  seats_taken =
    Repo.aggregate(
      from(e in Enrollment, where: e.course_id == ^id and is_nil(e.waitlisted_at)),
      :count
    )

  waitlist =
    from(e in Enrollment, where: e.course_id == ^id and not is_nil(e.waitlisted_at))
    |> Repo.all()
    |> Enum.map(&to_gleam_student/1)

  {:course, course.id, course.min_age, course.max_students, seats_taken, waitlist,
    course.waitlist_size}
end

# the enroll/2 function stays the same

Now let's write a failing test in Gleeunit:

pub fn waitlist_exceeded_test() {
  let course =
    Course(
      id: 2,
      min_age: 0,
      seats: 10,
      seats_used: 10,
      max_waitlist_size: 2,
      waitlist: [
        Student(id: 1, age: 22),
        Student(id: 2, age: 13),
        Student(id: 3, age: 54),
      ],
    )

  let student = Student(id: 4, age: 22)

  enrollment.enroll(student, course)
  |> should.equal(Rejected(NoSeats))
}

pub fn waitlisted_test() {
  let course =
    Course(
      id: 2,
      min_age: 0,
      seats: 10,
      seats_used: 10,
      max_waitlist_size: 2,
      waitlist: [],
    )

  let student = Student(id: 4, age: 22)

  enrollment.enroll(student, course)
  |> should.equal(Waitlisted)
}

One of these tests will accidentally pass (because we reused the NoSeats rejection reason, which might be a dubious choice). But the other one fails. Let's fix it now by actually implementing the waitlist. We need to import the gleam/list package on top and change the branch which previously resulted in NoSeats:

import gleam/list

# ...

pub fn enroll(student: Student, course: Course) -> EnrollmentDecision {
  case student.age >= course.min_age {
    False -> Rejected(AgeRequirementNotMet)
    True ->
      case course.seats > course.seats_used {
        False ->
          case list.length(course.waitlist) >= course.max_waitlist_size {
            True -> Rejected(NoSeats)
            False -> Waitlisted
          }
        True -> Enrolled
      }
  }
}

The final part will be done in Elixir. We have decided that a person should go to the waitlist, but now we have to persist it. Again, let's start with a test:

test "waitlist a student" do
  student1 = student_fixture()
  student2 = student_fixture()
  course = course_fixture(%{max_students: 1, waitlist_size: 10})
  Enrollment.enroll(course.id, student1.id)
  Enrollment.enroll(course.id, student2.id)
  assert Enrollment.waitlisted?(course.id, student2.id)
end

Then we need the waitlisted? function:

def waitlisted?(course_id, student_id) do
  from(e in Enrollment,
    where:
      e.student_id == ^student_id and
        e.course_id == ^course_id and not is_nil(e.waitlisted_at)
  )
  |> Repo.exists?()
end

Finally, we implement the infrastructure part, persisting the waitlist in the database. In the enroll function's case statement, we need to handle the :waitlisted return type case:

:waitlisted ->
  %Enrollment{}
  |> Enrollment.changeset(%{student_id: student_id, course_id: course_id, waitlisted_at: DateTime.utc_now()})
  |> Repo.insert()

All the tests pass now. We have, therefore, successfully implemented a waitlist function in our Gleam/Elixir application!

Looking at the requirements, we still have some way to go. We haven't touched on canceling an enrollment (when a user has successfully enrolled in the past but is no longer interested). Hint: this handles moving a person from a waitlist to a regular enrollment, so we at least need this type as a return value:

pub type CancellingEnrollmentDecision {
  Cancelled
  CancelledWithWaitlistProcessed(Student)
}

There's also a big topic that can be explored further but was not even covered in our requirements: how we should handle uniqueness. Everyone should be able to enroll in a given course only once, and they should not be able to cancel if they are not enrolled. There are at least three ways to model that behavior between Gleam and Elixir, but that's a topic for a separate article.

Why Did We Do It Again?

Everything we've done here could, of course, all have been written just in Elixir. The code would be shorter and in one language. In most cases, splitting the responsibilities between Elixir and Gleam probably would be a questionable choice. However, I personally think it's something worth considering if:

You like to model your application with types and are missing this from Elixir.
You expect your business logic to grow in complexity and want guarantees from a statically typed language to help you manage that complexity.
You want a strong separation between pure business logic and stateful application code.

If any of these are true for you, consider this arcane-looking setup. In any case, it's good to be aware that it is possible.

Wrapping Up

I have shown you how to call Gleam from an Elixir application and how to interpret the results. We wrote tests in both languages and some glue code.

Happy coding, whether you choose to do it only in Elixir or you also make use of Gleam!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Track Errors in Phoenix for Elixir with AppSignal

Roy Tomeij — 2024-07-09T00:00:00+00:00

AppSignal is a powerful error tracking and performance monitoring tool that can help you maintain reliability and speed in your Elixir applications.

In this tutorial, the first of a two-part series, you'll learn how to integrate AppSignal into your Elixir application, configure it for error tracking, interpret error reports, and leverage AppSignal's features to debug and resolve issues.

Whether you're a seasoned Elixir developer looking to improve your application's error handling or a beginner who wants to understand how error tracking works, this step-by-step tutorial will give you a firm foundation for using AppSignal to track errors in your Elixir app.

Prerequisites

To follow along with this tutorial, you need:

An Elixir development environment. If you don't have Elixir installed, follow the steps in this guide for your operating system.
An existing Elixir app. If you don't have one, go ahead and fork this app.
Some experience working with Elixir/Phoenix.

Let's get started!

Integrating AppSignal Into a Phoenix App

Quick tip: As of writing this article, AppSignal currently supports pure Elixir, Plug, and Phoenix apps. The examples used for this tutorial are centered around a Phoenix app featuring video games, users, and reviews, but the steps taken should be relatively similar for the other supported flavors.

AppSignal provides excellent support for the Phoenix framework and the installation process is very seamless. First, sign up for a free 30-day trial, no card details required.

Then open the Phoenix app's dependencies file and add AppSignal's Phoenix package:

# mix.exs

...
defp deps do
  [
    ...
    {:appsignal_phoenix, "~> 2.0"}
  ]
end
...

Run mix deps.get to add the package to your app. Once it's added successfully, install it by running the command below:

mix appsignal.install <PUSH_API_KEY>

Note: You can get your PUSH_API_KEY from the AppSignal dashboard.

When you run this command, you'll be presented with some options to customize the installation:

Validating Push API key: Valid! 🎉
What is your application's name? [game_reviews_app]: <APP NAME>

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

The first option is your app's name as you want it to appear on the AppSignal dashboard.
The second option is how to configure AppSignal: via a configuration file or a system variable.

Configuring AppSignal

For your app to send data to AppSignal, you should have a minimal configuration in config/appsignal.exs, as shown below:

# config/appsignal.exs

import Config

config :appsignal, :config,
  otp_app: :game_reviews_app,
  name: "Phoenix API app",
  push_api_key: <AppSignal Push API key>,
  env: Mix.env

Or you can set this up using system environment variables. One thing to note is that this file can be configured to your liking using a variety of options.

Assuming the configuration proceeds without a hitch, you should now see your app automatically added to the AppSignal dashboard, as shown below:

Note: It may take a couple of minutes for your app's data to start showing.

Great, now let's see how we can instrument an error in AppSignal.

Instrumenting Elixir Errors

This example app is a simple Phoenix LiveView app featuring a video game and reviews.

Once you've cloned the application to your development environment, run mix deps.get to install all the dependencies, then you can run the app with:

mix phx.server

This will make the app available on http://localhost:4000.

Go ahead and test it by creating a few games with the following command:

curl --location --request POST 'http://localhost:4000/api/games' \
--header 'Content-Type: application/json' \
--header 'Accept: */*' \
--header 'Host: localhost:4000' \
--header 'Connection: keep-alive' \
--data-raw '{
    "game": {
        "title": "Hell divers",
        "genre": "Action",
        "publisher": "Arrowhead Studios"
    }
}'

Having made a couple of games this way, you can now make a GET request for the games list (I use Insomnia for this, but you can use whatever you like):

Which should execute successfully, as you can see above.

Now, let's deliberately cause an error to see how it will be shown on the AppSignal dashboard.

Firstly, create a function defining a custom exception:

# lib/exceptions/plug_exception.ex

defmodule GameReviewsApp.PlugException do
  defexception [:plug_status, message: ""]
end

Then, modify the index action in the games controller to call this function:

defmodule GameReviewsAppWeb.GameController do
  ...
  def index(conn, _params) do
    raise GameReviewsApp.PlugException, plug_status: 403, message: "You're not allowed!"
    games = Games.list_games()
    render(conn, :index, games: games)
  end
  ...
end

Now when we make a request to the games list, this custom exception is raised:

And the same is shown on the AppSignal dashboard:

If you click on the displayed error link, AppSignal gives you more context on the error, which should help you resolve it:

Next up, let's learn how we can track Ecto queries inside AppSignal.

Instrumenting Ecto queries in AppSignal

Ecto is the most popular database abstraction library for Elixir apps.

AppSignal provides native support for Ecto by hooking into the Ecto query telemetry layer, giving you query profiling, identification of slow queries, and more, with no extra work on your part (as long as the otp_app name within the AppSignal configuration file matches the name of your app in config.exs):

# config/config.exs

...
# General application configuration
import Config

config :game_reviews_app,
...

# config/appsignal.exs

import Config

config :appsignal, :config,
  otp_app: :game_reviews_app,
  ...

With our configuration done, AppSignal's instrumentation will now automatically pick up Ecto queries and display them on this dashboard:

Now this might be good enough for normal Ecto queries. But to pick up query types such as preloads in the right way, you'll need to edit your app's repo:

# lib/game_reviews_app/repo.ex

defmodule PhxRestApiApp.Repo do
  # use Ecto.Repo,       #...replace this default line with the one below...
  use Appsignal.Ecto.Repo,
    otp_app: :game_reviews_app,
    adapter: Ecto.Adapters.SQLite3
end

Next, let's learn how to track Phoenix LiveViews using AppSignal.

Instrument Phoenix LiveView with AppSignal

With the latest AppSignal for Elixir package (version 2.12.0 as of writing), you can instrument LiveView dashboards in AppSignal.

Simply adding the package is enough to automatically add your app's live views using the AppSignal package's telemetry handlers. Or you can configure the instrumentation manually using helpers made available by the package.

The code below shows how easy it is to manually configure LiveView instrumentation by adding a single line within application.ex:

# lib/game_reviews_app/application.ex

...
def start(_type, _args) do
  Appsignal.Phoenix.LiveView.attach() # <= ...added this line

  children = [
    ...
  ]
  ...
end
...

And with that, you're able to view LiveView mount, handle_event, and handle_param events in the AppSignal dashboard, like so:

Additionally, with the AppSignal LiveView instrumentation helpers, you can wrap LiveView action definitions within an instrument/3 block for even more granular visualizations, like so:

# live/welcome_live.ex

defmodule GameReviewsAppWeb.WelcomeLive do
  use GameReviewsAppWeb, :live_view

  import Appsignal.Phoenix.LiveView, only: [instrument: 4]

  def mount(_params, _session, socket) do
    instrument(__MODULE__, "timer instrumentation", socket, fn ->
      :timer.send_interval(1000, self(), :tick)
      {
        :ok,
        assign(socket, current_time: DateTime.utc_now())
      }
    end)
  end

  def render(assigns) do
    ~H"""
    <div class="container">
      <h1>Welcome to my LiveView App</h1>
      <p>Current time: <%= @current_time %></p>
    </div>
    """
  end
end

When sending metrics to AppSignal, you can use a gauge, a counter, or a distribution.

In the code above, we define a counter measurement called "timer instrumentation" to send current time measurements to the AppSignal dashboard continuously.

As you can see below, by switching the dashboard view from "web" to "live_view", "timer instrumentation" is now available on the dashboard as a sample (with all the details you need to dig further):

Finally, I'll highlight a nifty feature that makes AppSignal really stand out in the application performance monitoring space: automated dashboards.

AppSignal's Automated Dashboards for Elixir

Whenever you add AppSignal to an app built using Elixir, it automatically creates a dashboard.

When we add the AppSignal package to the Phoenix app, we get an automated dashboard and notification — in this case, an Erlang Virtual Machine dashboard:

Check out the AppSignal for Elixir documentation for more information.

Wrapping Up

In this article, we've seen how to track Ecto queries and Phoenix LiveViews using AppSignal for Elixir. Consider this an introduction to what is possible when it comes to error tracking in Elixir with AppSignal.

Next time, we'll deep dive into custom instrumentation using Phoenix and Ecto's in-built telemetry features, learning how you can craft and send custom metrics to AppSignal for monitoring even more powerful use cases.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Leverage Concurrency Efficiently When Managing Multiple Tasks in Elixir

Roy Tomeij — 2024-06-25T00:00:00+00:00

When working on multiple tasks, it's important to consider performing them concurrently. However, when using concurrency, we need to be careful not to overload our system resources.

In this article, we will cover the following:

What is concurrency?
How can we use concurrency in Elixir?
Common issues when working with a series of tasks concurrently.
How to efficiently manage a series of tasks using the Task module.

Let's get going!

What is Concurrency?

Concurrency is the ability to execute multiple tasks at the same time.

Concurrency and parallelism are crucial features in modern programming. The operating systems installed on our computers are designed to incorporate these features. Both concurrency and parallelism involve executing multiple tasks simultaneously. However, concurrency focuses on managing multiple tasks within one resource. This can be achieved through techniques like multitasking and asynchronous programming. In an operating system, concurrency can be achieved by opening multiple applications to perform different tasks at the same time, also referred to as multitasking.

On the other hand, parallelism involves executing multiple tasks simultaneously across multiple processing units (such as multiple CPU cores or distributed computing resources), thus improving system speed.

How Can We Use Concurrency in Elixir?

To achieve concurrency in Elixir, you need to initiate a process and execute the function or code responsible for performing a specific task within that process.

Here's an example of some basic Elixir code that we can take a look at.

def send_message(phone_number) do
  # Simulate some processing time
  Process.sleep(3000)
  IO.puts("Sending message to #{phone_number}")
  {:ok, "sent"}
end

In this example, we've added a 3-second sleep to simulate the fact that sending a message to an external phone number can sometimes cause some latency.

When send_message/1 is invoked, this function will delay processing for 3 seconds before printing a message. It will also return an {:ok, "sent"} tuple to indicate that the message was successfully sent to the given phone number.

When executing synchronous Elixir code like the code above, we must wait for it to complete before receiving the output. If we need to send messages to multiple phone numbers, each with a delay of 3 seconds, it could take a significant amount of time to receive a notification that all messages have been successfully sent.

Taking It One Step Further

Now, let's take it a step further and write some code that sends a message to multiple phone numbers.

def send_to_all(phone_numbers) do
  Enum.each(phone_numbers, &send_message/1)
end

The send_to_all/1 function takes a list of phone numbers as an argument and uses the Enum.each/2 function to iterate over the list, sending a message to each phone number using the send_message function.

If you run this function in iex, you will notice two things:

It takes longer to receive the final output, which is an atom :ok. Enum.each/2 returns :ok when invoked.
Printing happens one at a time, with each print taking 3 seconds. Depending on the number of phone numbers provided, send_message/1 is called 5 times in our case, taking about 15 seconds to complete.

This means that when sending a message to a given phone number, each phone number has to wait for the preceding phone number to execute before moving on to the next one. The execution is synchronous, also known as blocking code.

Elixir provides the ability to run code concurrently or asynchronously. Asynchronous code allows you to execute multiple tasks simultaneously. This type of code is also referred to as non-blocking code because it does not hinder the primary execution of a program.

Let's improve the send_to_all/1 function by running the task of sending messages to each phone number asynchronously. We need to start a process for each task.

A process is a separate entity where code execution occurs. These processes are lightweight, run concurrently, and are isolated from each other. The code we write runs inside processes. The IEx (interactive shell) is an example of an Elixir process.

def send_to_all(phone_numbers) do
  Enum.each(phone_numbers, fn phone_number ->
    Task.start(fn -> send_message(phone_number) end)
  end)
end

Here, we've used Task.start/1 (which we'll go over in more detail later) to start a process for each task. After we recompile and run this function again in our IEx, you will notice two things:

It runs faster than before because it's now running asynchronously. The final output — :ok — is immediately returned even before we see the printed success messages.
It instantly prints the success message at once for all tasks, which means that all the messages are being sent at the same time.

Running Tasks Concurrently using the Task Module

Elixir has a standard library Task module commonly used for running code concurrently. This allows you to perform tasks asynchronously, which can lead to improved performance and responsiveness in your applications.

Let's go back to our send_to_all/1 function. At first, we used this function to send messages to multiple phone numbers sequentially.

# Sequential version of send_to_all/1
def send_to_all(phone_numbers) do
  Enum.each(phone_numbers, fn phone_number ->
    Task.start(fn -> send_message(phone_number) end)
  end)
end

In the sequential version, send_message/1 was called for each message in the list, one after the other.

Later on, we converted the sequential code into concurrent code using Task.start/1.

# Asynchronous version using Task.start/1
def send_to_all(phone_numbers) do
  Enum.each(phone_numbers, fn phone_number ->
    Task.start(fn -> send_message(phone_number) end)
  end)
end

In the asynchronous version, Task.start/1 creates a separate task for each message, allowing the tasks to execute concurrently.

The Task module has several useful functions to run code concurrently. Let's now look at a few of these functions.

`Task.start/1`

Task.start/1 is used to start a new process/task. It takes in an anonymous function with zero arity as an argument, where the intended work is performed. Usually, it doesn't return the result of the executed function. Instead, it returns an {:ok, PID} tuple, where :ok means the process was started successfully, and PID stands for process identifier: a number that uniquely identifies an Elixir process.

Let's try it out in IEx:

iex > {:ok, pid} = Task.start(fn ->  IO.puts("converting to asyncronous code") end)
         converting to asyncronous code
          {:ok, #PID<0.153.0>}

The message is printed instantly and the result is {:ok, #PID<0.153.0>}. This means that the process was started successfully. pid represents the process started by Task.start/1.

However, it's hard to know if all the messages were sent successfully using Task.start/1, since it will still return {:ok, PID}, which stands for a process that's started successfully.

It is useful in cases where you don't have interest in the returned result or whether it completes successfully.

A good example is working on a background job processing system, such as sending messages to registered users every midnight. You can use Task.start/1 to asynchronously handle each task that sends messages to users. In this context, you're not particularly interested in the return value of Task.start/1, because your main concern is to handle the tasks asynchronously, and you have a mechanism to track the status of each task. This mechanism can involve saving the status in the database, ensuring that any pending tasks are retried in subsequent runs.

In case something goes wrong when sending messages to users, the status of the message will still be pending in the database. You can guarantee that the worker will run again at midnight and retry for pending messages.

`Task.async/1`

Like Task.start/1, Task.async/1 is also used to start a process. The difference between Task.async/ and Task.start/ is that with Task.async/1, you can retrieve the function result executed within the process. It takes in an anonymous function with zero arity which starts a process and then returns a %Task{} struct.

Let's try it out in IEx:

iex > Task.async(fn -> IO.puts("converting to asyncronous code using async") end)
    converting to asyncronous code using async
      %Task{
        mfa: {:erlang, :apply, 2},
        owner: #PID<0.109.0>,
        pid: #PID<0.110.0>,
        ref: #Reference<0.0.13955.2933390707.3162308609.257437>
      }

The message is printed immediately and returns a %Task{} struct as the output. The returned Task struct includes useful information about the process that's started:

owner represents the Process Identifier (PID) of the process that started the Task. In our example, the owner is IEx since we are running code within the shell. If we run self(), the function that returns the PID of the current process, then the PID returned by self() will indeed match the PID of the owner of the task started.
pid: The process identifier of the process started by the task. You can use this PID to monitor the process further if needed.
ref: This is a process monitor reference that can be used in cases where you want to be notified about how and where a process exits.

To retrieve a result from the returned task, use either Task.await/2 or Task.yield/2. They both accept a %Task{} struct as an argument and handle process timeout. However, they handle the timeout differently.

Task.await/2 and Task.yield/2 have a default timeout of 5000ms or 5 seconds to ensure that processes don't get stuck waiting for a result forever. Task.await/2 will cause an exception and crash the process if there is a slower task taking more than 5000ms to complete, while Task.yield/2 returns nil.

Let's modify send_to_all/1 to use Task.async/1, then retrieve its result using Task.await/2. Later, use Task.yield/2:

def send_to_all(phone_numbers) do
  phone_numbers
  |> Enum.map(fn phone_number ->
    Task.async(fn -> send_message(phone_number) end)
  end)
  |> IO.inspect(label: "started tasks++++")
  |> Enum.map(fn task -> Task.await(task) end)
end

send_to_all/1 iterates over a given list of phone numbers using Enum.map/2 instead of Enum.each/2 to retrieve the result of each task, and spawns a task for each using Task.async/1. Then we use Task.await/2 to retrieve the result of each task. It waits for the task to complete and then returns a result.

Run the send_to_all/1 function in IEx:

iex > send_to_all(1..10)
started tasks++++[
  %Task{
    mfa: {:erlang, :apply, 2},
    owner: #PID<0.150.0>,
    pid: #PID<0.151.0>,
    ref: #Reference<0.0.19203.3913757788.1002242054.113465>
  },
  %Task{
    mfa: {:erlang, :apply, 2},
    owner: #PID<0.150.0>,
    pid: #PID<0.152.0>,
    ref: #Reference<0.0.19203.3913757788.1002242054.113469>
  },..
]

sending message  to 1
sending message  to 2
sending message  to 3
sending message  to 4
sending message  to 5
sending message  to 6
sending message  to 7
sending message  to 8
sending message  to 9
sending message  to 10

[
  {:ok, "sent"},
  {:ok, "sent"},
...
]

Looking at the inspected result, each task represents a spawned process. They all have a different pid with a similar owner pid meaning that the caller process (IEx) is one. Once each task is completed, the messages are printed at the same time and the result returned. The final result returned is a list of output expected from send_message/1.

Now, let's retrieve the result from each spawned task using Task.yield/2:

def send_to_all(phone_numbers) do
  phone_numbers
  |> Enum.map(fn phone_number ->
    Task.async(fn -> send_message(phone_number) end)
  end)
  |> IO.inspect(label: "started tasks++++")
  |> Enum.map(fn task -> Task.yield(task) end)
end

iex > send_to_all(1..10)
started tasks++++[
  %Task{
    mfa: {:erlang, :apply, 2},
    owner: #PID<0.150.0>,
    pid: #PID<0.151.0>,
    ref: #Reference<0.0.19203.3913757788.1002242054.113465>
  },
  %Task{
    mfa: {:erlang, :apply, 2},
    owner: #PID<0.150.0>,
    pid: #PID<0.152.0>,
    ref: #Reference<0.0.19203.3913757788.1002242054.113469>
  },..
]

sending message  to 1
sending message  to 2
sending message  to 3
sending message  to 4
sending message  to 5
sending message  to 6
sending message  to 7
sending message  to 8
sending message  to 9
sending message  to 10

[
  :ok, {:ok, "sent"},
  :ok, {:ok, "sent"},
...
]

The result is similar. However, Task.yield/2 returns an {:ok, term()} tuple, where term represents the result returned by the function executed within the process. That's why our output is a list of the {:ok, {:ok, "sent"}} tuple where {:ok, "sent"} is the expected result from send_message/1.

In our send_to_all/1 function, we used Task.start/2 and Task.async/2 to start each task of sending a message to a given phone number in the background, allowing them to run concurrently. Starting a separate process for each task when sending a message ensures its isolation. Therefore, if one task encounters an error or fails for any reason, it won't affect the execution of other tasks.

An Example with an Error

For example, let's assume we have introduced an error in one of the tasks.

def send_message(3) do
  # Simulate some processing time
  Process.sleep(3000)
  IO.puts("Sending message to #{3 + "a"}")
  {:ok, "sent"}
end

def send_message(phone_number) do
  # Simulate some processing time
  Process.sleep(3000)
  IO.puts("Sending message to #{phone_number}")
  {:ok, "sent"}
end

# Asynchronous version using Task.start/1
def send_to_all(phone_numbers) do
  Enum.each(phone_numbers, fn phone_number ->
    Task.start(fn -> send_message(phone_number) end)
  end)
end

Here, the send_message/1 function has two function heads. The first function head pattern matches on 3 and deliberately raises an arithmetic error by adding the number 3 to a string "a". The second function head handles other phone numbers values and simulates sending messages as before.

iex > send_to_all(1..5)
06:19:17.562 [error] Task #PID<0.294.0> started from #PID<0.291.0> terminating
** (ArithmeticError) bad argument in arithmetic expression
    :erlang.+(3, "a")

[
  ok: #PID<0.292.0>,
  ok: #PID<0.293.0>,
  ok: #PID<0.294.0>,
  ok: #PID<0.295.0>,
  ok: #PID<0.296.0>
]
sending message  to 1
sending message  to 2
sending message  to 4
sending message  to 5

If you pass a list of phone numbers to send_to_all/1 and one of them is 3, the corresponding task will raise an arithmetic error, but other tasks will continue to run. From the results, we can see that we are notified of an arithmetic error in one of the started processes (#PID<0.294.0>). We also receive printed messages indicating that the messages were sent successfully to numbers 1, 2, 4, and 5. That means that the task started for number 3 encountered an error, but didn't prevent other processes from performing their tasks.

This approach enhances concurrency by allowing tasks to run concurrently, while also ensuring that they are isolated processes. This helps to create a more robust system.

A Problem: Working with a Series of Tasks

We've seen how we have been able to send messages to the given phone numbers concurrently using Task.async/1 and Task.start/1. In the examples provided, send_to_all/1 spawned a process for each task when sending a message. Even though our aim is to leverage concurrency and build a fault-tolerant and robust application, we should also take into consideration that processes don't share memory.

The more we increase phone numbers, the more we start a high number of concurrent tasks, each one occupying their own memory. Therefore, increasing the number of concurrent tasks can lead to an application consuming a significant amount of memory. That, in turn, can cause slow application performance and a spike in system usage, degrading system performance and maybe even making other services unresponsive.

Let's make send_to_all/1 send messages to 1 million phone numbers. But first, we'll modify it by renaming send_to_all/1 to differentiate the send_to_all/1 using Task.async/1 from the one using Task.start/1.

defmodule Sender do
  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent"}
  end

  # Asynchronous version using Task.start/1
  def send_to_all_start(phone_numbers) do
    Enum.each(phone_numbers, fn phone_number ->
      Task.start(fn -> send_message(phone_number) end)
    end)
  end

  # Asynchronous version using Task.async/1
  def send_to_all_async(phone_numbers) do
    phone_numbers
    |> Enum.map(fn phone_number ->
      Task.async(fn -> send_message(phone_number) end)
    end)
  end
end

In this modification, send_to_all_async/1 and send_to_all_start/1 functions have been defined to differentiate the behavior of Task.async/1 and Task.start/1 when attempting to run a million processes.

Now, open IEx (or recompile it using the recompile() command if it was already running) and call send_to_all_start/1 with a million phone numbers. We're not using a real phone number, therefore we simply pass a 1..1_000_000 range of numbers.

iex > Sender.send_to_all_start(1..1_000_000)

06:12:14.067 [error] Too many processes
** (SystemLimitError) a system limit has been reached
    :erlang.spawn(Task.Supervised, :noreply, [{:nonode@nohost, #PID<0.150.0>, #PID<0.150.0>}, [#PID<0.150.0>, #PID<0.142.0>], [#PID<0.150.0>], {:erlang, :apply, [#Function<7.61290973/0 in Sender.send_to_all_start/1>, []]}])
    (elixir 1.15.5) lib/task/supervised.ex:6: Task.Supervised.start/3
    (elixir 1.15.5) lib/enum.ex:4356: Enum.map_range/4
    (elixir 1.15.5) lib/enum.ex:4356: Enum.map_range/4
    (elixir 1.15.5) lib/enum.ex:4356: Enum.map/2
    iex:2: (file)

The first thing we can see is an error message notifying us that too many processes have been started, exceeding the system limit. This is happening as soon as Task.start/1 is invoked for each phone number in the range from 1 to 1,000,000. Success messages have been printed, but they are not showing for all of the phone numbers. The highest number of messages that I can see in the output is 262073. Since the numbers are not sorted, it is easy to miss the highest number. However, the total number of messages sent is around 262000.

What's Going On Here and How Do We Fix It?

We are having issues with starting a million processes because the number of tasks spawned by Task.start/1 exceeds the default limit of processes (262144, for performance and memory-saving reasons) that can be started in BEAM. However, we can override this limit using +P NUM. This means we can increase the limit by running the command iex --erl '+P 1000000'.

To apply this change, close IEx and restart it by running the command iex --erl '+P 1000000' -S mix. Then, you can call send_to_all_start/1 again, with a range of phone numbers from 1 to 1000000.

sender tracey$ iex --erl '+P 1000000' -S mix
iex > Sender.send_to_all_start(1..1_000_000)
 :ok

This time, the function Enum.each/2 has returned the final output :ok. Additionally, success messages have been printed for all of the one million phone numbers. It is good that we can override the default system limit, but we must be cautious when increasing the limit, as it can cause the VM to use more memory, potentially leading to performance issues.

To monitor memory usage before and after a system limit increase, use the :observer.start command in IEx to open the Observer tool. The Observer shows a spike in memory usage when the system limit is increased.

Before the system limit increase:

After the system limit increase:

Sending Messages To the Phone Numbers with `Task.async/1`

Next, we will use the send_to_all_async/1 function to send messages to a million phone numbers. This function uses Task.async/1 to start a process for each task.

iex > Sender.send_to_all_async(1..1_000_000)

With Task.async/1, you can send a message to all the phone numbers. However, this capability may not be available on all machines, so keep that in mind.

The Observer shows a spike in memory usage when using Task.async/1.

Running these tasks synchronously works out better in terms of memory usage compared to running them asynchronously when using a combination of Enum with either Task.async/1 or Task.start/1.

When using Task.start/ or Task.async/1 to start a process for each task, you're essentially creating a separate process for each message you want to send. While this can be an efficient way to handle concurrency, it can also lead to increased memory usage, especially when dealing with a large number of tasks simultaneously.

The spike in memory usage we're seeing in the Observer is likely due to the fact that each process created by Task.async/1 and Task.start/1 consumes memory. With a million phone numbers, you're creating a million processes, which can quickly exhaust available memory, especially if each process is doing significant work or holding onto a large amount of data.

The Solution: Use `Task.async_stream/3`

When utilizing Task processes, the goal is to achieve concurrency to efficiently send messages to designated phone numbers. However, it is important to be mindful of system resources and avoid sudden increases in pressure.

Fortunately, the Task module provides a useful feature known as Task.async_stream/3. It works similarly to Enum.map/2 and Task.async/2 combined, as it creates task processes from a given list of items.

With Task.async_stream/3, you can perform the task for each item in the list concurrently, by starting a process. The only difference is that you can set a limit on the number of processes running at the same time with this function.

For instance, suppose the message application needs to send messages to 100 phone numbers using Task.async_stream/3. In this case, we can set the concurrency limit to 5, which means that, at most, only 5 processes will start to send messages to the given phone numbers at the same time.

Task.async_stream/3 takes in three arguments:

Enumerable: This argument represents the collection of items that you want to process concurrently. It can be an Enum or a Stream.
Anonymous function: This must take a single argument that represents an element of the enumerable. The function is applied to each enumerable element. It defines the task to be executed concurrently for each item.
Options, used to control the level of concurrency, the time tasks are allowed to run, ordering of the results and the action to take when a task times out.

Task.async_stream/3 returns a stream, which is a lazy enumerable. Therefore, transformations or computations on the stream are not performed as soon as the stream is created. Instead, they are performed until the stream is explicitly consumed or operated upon.

An Example Using `Task.async_stream/3`

Here's an example of using Task.async_stream/3 in IEx:

iex > Task.async_stream(1..5, fn phone_number -> send_message(phone_number) end)
#Function<3.112246596/2 in Task.build_stream/3>

The result that's returned is a stream. To run this stream, we can use Stream.run/1, which returns :ok and isn't useful when we're not interested in the final result. Another alternative is to use the Enum functions. This is useful when you intend to perform other tasks with the expected result.

iex > 1..5 |> Task.async_stream(fn phone_number -> send_message(phone_number) end) |> Stream.run()
sending message  to 1
sending message  to 2
sending message  to 3
sending message  to 4
sending message  to 5
:ok

Stream.run/1 is used to run the stream without collecting the result. The messages are sent to the given phone numbers and once all tasks are completed, :ok is returned.

iex > 1..5 |> Task.async_stream(fn phone_number -> send_message(phone_number) end) |> Enum.to_list
sending message  to 1
sending message  to 2
sending message  to 3
sending message  to 4
sending message  to 5

[
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"}
]

Enum.to_list/1 runs the stream and collects the results into a list.

Task.async_stream/3 accepts a list of options. When not explicitly passed, the options default to their default values. Let's dive further into the options passed to Task.async_stream/3.

`:max_concurrency`

:max_concurrency is responsible for setting a limit on the number of processes running at the same time. Its value defaults to the number of logical cores available in a system.

When running Task.async_stream/3 in IEx with a 1..20 range of phone numbers:

iex >  Task.async_stream(1..20, fn phone_number -> send_message(phone_number) end) |> Stream.run()

I noticed that the number of items processed at the same time was in a batch of 10 until completion. This number can differ depending on the machine used, so don't be surprised if you're seeing a batch of 20 or even 5 items processed simultaneously in your machine. This is because of the logical cores available in different machines. To confirm the number of logical cores available, use System.schedulers_online/0. For machines whose CPU has less than 4 logical cores, Task.async_stream/3 will appear to be slower.

We can emulate the slow performance by setting :max_concurrency to a value less than 4.

iex >  Task.async_stream(1..20, fn phone_number -> send_message(phone_number) end, max_concurrency: 2) |> Stream.run()

When the concurrency limit is set to 2, it takes more time to complete compared to using the default value set, 10. This is due to the fact that with a lower concurrency limit, fewer tasks are processed concurrently.

When setting max_concurrency, consider your system resources. Verify that the system has enough resources to handle the specified concurrency limit effectively and also bear in mind the need for performance. Depending on what works for your use case, you can decrease or increase the concurrency limit.

Our application aims to send messages to a million phone numbers. At present, the max_concurrency setting on my machine is set to 10 and is working satisfactorily. However, it is taking a long time to process all the messages. After 17 minutes, only about 3,000 messages have been sent. To improve the processing speed, we need to increase the concurrency limit. I shall set it to 100, allowing more items to be processed concurrently, while also ensuring we do not overload the system.

iex > 1..20
    > |> Task.async_stream( fn phone_number -> send_message(phone_number) end, max_concurrency: 100)
    > |> Stream.run()

After increasing the max_concurrency to 100:

In 4 minutes, about 6,000 messages were sent.
There is no spike in memory allocation when we open the observer.

Here's the observer memory allocation when max_concurrency is set to 10:

And here it is when max_concurrency is set to 100:

`:ordered`

When we execute this code in IEx, you will notice that the results are returned in the same order as the input data:

iex > 1..5
    > |> Task.async_stream( fn phone_number -> send_message(phone_number) end, max_concurrency: 100)
    > |> Enum.to_list

sending message  to 1
sending message  to 2
sending message  to 3
sending message  to 4
sending message  to 5

[
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"},
:ok, {:ok, "sent"}
]

This is because we have the :ordered option set to true by default in the Task.async_stream/3 function.

However, setting :ordered to true can result in slower processing if a task takes longer to complete. This is because Task.async_stream/3 will wait for a slow task to finish before moving on to the next one. Therefore, it is important to consider the trade-off between ordered results and processing speed when using this function.

To improve processing speed, you can disable ordering by setting the :ordered option to false. This way, Task.async_stream/3 won't wait for a slow task to complete processing before moving on to the next task.

Let's introduce a slow process in our Sender application.

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(4000)
    IO.puts("Sending message to 3")

    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent", phone_number}
  end

  # Asynchronous version using Task.async_stream/3
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Task.async_stream(fn phone_number -> send_message(phone_number) end)
    |> Enum.to_list()
  end
end

The send_message/1 function accepts one argument, a phone number. If the phone number is 3, the function simulates a slow process by sleeping for 4 seconds. Otherwise, it simulates a regular process by sleeping for 3 seconds. The output result includes the phone number that's being processed, which allows us to check the order of the returned result.

iex > Sender.send_to_all(1..5)

sending message  to 1
sending message  to 2
sending message  to 4
sending message  to 5
sending message  to 3

[
{:ok, {:ok, "sent"}, 1},
{:ok, {:ok, "sent"}, 2},
{:ok, {:ok, "sent"}, 3},
{:ok, {:ok, "sent"}, 4},
{:ok, {:ok, "sent"}, 5}
]

The print message of number 3 is displayed last, indicating that it took longer to process before sending. However, the collected list is returned in the same order as the given input. This means that, while number 3 took longer to complete, numbers 4 and 5 had to wait before being operated on.

In our case, we want to check whether the status of the messages sent to the given phone numbers was successful or not. To speed up things, let's disable the :ordered option by setting it to false, since we don't care about the order.

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(4000)
    IO.puts("Sending message to 3")

    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent", phone_number}
  end

  # Asynchronous version using Task.async_stream/3
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Task.async_stream(fn phone_number -> send_message(phone_number) end, ordered: false)
    |> Enum.to_list()
  end
end

iex > Sender.send_to_all(1..5)

sending message  to 1
sending message  to 2
sending message  to 4
sending message  to 5
sending message  to 3

[
{:ok, {:ok, "sent"}, 1},
{:ok, {:ok, "sent"}, 2},
{:ok, {:ok, "sent"}, 4},
{:ok, {:ok, "sent"}, 5},
{:ok, {:ok, "sent"}, 3},
]

The number 3 is collected last in the returned result. This way, we can be sure that Task.async_stream/3 won't be idle waiting for the processing of number 3 to complete, before moving to the next.

`:timeout`

The Task.async_stream/3 function has a :timeout option which sets a limit on how long each task can run. By default, the timeout is set to 5000 milliseconds or 5 seconds. You can specify the time limit in milliseconds or set it to :infinity if you want to allow the task to run indefinitely. If a task takes longer than the specified timeout, it will raise an exception and terminate the current process.

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(7000)
    IO.puts("Sending message to 3")

    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent", phone_number}
  end

  # Asynchronous version using Task.async_stream/3
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Task.async_stream(fn phone_number -> send_message(phone_number) end, ordered: false)
    |> Enum.to_list()
  end
end

Here, the processing time of the slow process/task is increased to 7000ms, which exceeds the default timeout in Task.async_stream/3.

iex > Sender.send_to_all(1..5)

sending message  to 1
sending message  to 2
sending message  to 4
sending message  to 5

** (exit) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out
    (elixir 1.15.5) lib/task/supervised.ex:314: Task.Supervised.stream_reduce/7
    (elixir 1.15.5) lib/enum.ex:4387: Enum.reverse/1
    (elixir 1.15.5) lib/enum.ex:3704: Enum.to_list/1
    iex:3: (file)

When we execute the Sender.send_to_all(1..5) function in IEx, an exception is raised, which stops the stream and crashes the current process. This is because the slow task takes more than 5000ms to complete, while Task.async_stream/3 has a timeout limit of 5000ms.

`:on_timeout`

It's difficult to predict the duration of a task, as several factors can contribute to its slow processing. For instance, processing a large amount of data can be time-consuming, as can waiting for a slow third-party API to respond.

Instead of allowing Task.async_stream/3 to raise an exception that stops the stream, we can use the :on_timeout option by setting it to :kill_task. This option determines the action to take when the task times out. Setting it to :kill_task will cause Task.async_stream/3 to ignore the process that exits with a timeout and continue with other tasks.

By default, it's set to :exit. This results in the task exceeding the timeout to stop the stream and crash the current process, as seen with the :timeout option.

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(7000)
    IO.puts("Sending message to #{3}")
    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent", phone_number}
  end

  # Asynchronous version using Task.async_stream/3
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Task.async_stream(fn phone_number -> send_message(phone_number) end,
      ordered: false,
      on_timeout: :kill_task
    )
    |> Enum.to_list()
  end
end

Add the :on_timeout option and set it to :kill_task in Task.async_stream/3. Then, execute Sender.send_to_all(1..5) in IEx. We can observe that the other tasks are processed completely, while the slow process is ignored, and returns {:exit, :timeout} instead of crashing.

iex > Sender.send_to_all(1..5)

sending message  to 1
sending message  to 2
sending message  to 4
sending message  to 5

[
{:ok, {:ok, "sent"}, 1},
{:ok, {:ok, "sent"}, 2},
{:ok, {:ok, "sent"}, 4},
{:ok, {:ok, "sent"}, 5},
exit: :timeout
]

Before we wrap up, let's take a very quick look at monitoring memory using Observer.

Monitor Memory with Erlang Observer

Let's use Erlang Observer to monitor memory allocation. We can compare memory usage when defining the send_to_all/1 function using Task.async_stream/3 and using a combination of Enum.map/2 and Task.async/1.

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(7000)
    IO.puts("Sending message to #{3}")
    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent", phone_number}
  end

  # Asynchronous version using Task.async_stream/3
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Task.async_stream(fn phone_number -> send_message(phone_number) end,
      max_concurrency: 100,
      ordered: false,
      on_timeout: :kill_task
    )
    |> Stream.run()
  end
end

defmodule Sender do
  def send_message(3) do
    # Simulate slow processing time
    Process.sleep(7000)
    IO.puts("Sending message to #{3}")
    {:ok, "sent", 3}
  end

  def send_message(phone_number) do
    # Simulate some processing time
    Process.sleep(3000)
    IO.puts("Sending message to #{phone_number}")
    {:ok, "sent"}
  end

  # Asynchronous version using Task.async/1
  def send_to_all(phone_numbers) do
    phone_numbers
    |> Enum.map(fn phone_number ->
      Task.async(fn -> send_message(phone_number) end)
    end)
  end
end

The combination of Task.async/1 and Enum.map/2 can significantly impact memory consumption compared to using Task.async_stream/3.

Task.async_stream/3 is a powerful tool for effectively utilizing concurrency without overloading system resources when working with a series of tasks.

Here are a couple of benefits of using Task.async_stream/3:

It prevents sudden spikes in system usage: By setting the concurrency limit option, you can control the number of concurrent tasks being executed at any given time. This helps prevent the system from becoming overwhelmed with too many tasks running simultaneously, which could lead to spikes in resource usage, such as memory or CPU.
It handles back pressure: Back pressure occurs when there's resistance to the progress of turning input to output. By setting the concurrency limit, Task.async_stream/3 ensures that tasks are processed at a manageable rate, preventing resource exhaustion.

And that's it!

Wrapping Up

Elixir has several tools that are useful when it comes to leveraging concurrency, and Task processes is one of them.

In this post, we explored the concept of concurrency and some of the issues we might face when dealing with a series of tasks while using concurrency.

We then explored the use of Task.async_stream/3, a Task module function that is effective in handling a large number of tasks, as it provides both concurrency and performance.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Advanced Dependency Injection in Elixir with Rewire

Roy Tomeij — 2024-06-11T00:00:00+00:00

In our last post, we explored how Dependency Injection (DI) is a powerful design pattern that can improve our ExUnit tests.

In this article, we will dive deeper into the topic of DI in Elixir, focusing on the Rewire library for Elixir projects.

We will cover Rewire's core concepts, how to get started with it, and practical examples. We will also see how to use Rewire alongside Mox.

Let's get started!

Introduction to Rewire

One of the challenges we faced in our previous article was the lack of a structured way to define and inject dependencies into our modules. We had to manually define our mocks for testing.

This is where Rewire and Mox come into play:

Rewire provides a more structured and flexible way to implement DI in Elixir projects.
Mox is a library that allows us to define mocks for our tests.

Combining these two tools can significantly improve the testability and modularity of our Elixir applications.

Let's get started by setting up a sample project that leverages both libraries.

Why Use Rewire and Mox for Elixir?

To recap part one of the series, we discussed the benefits of DI for testability and modularity. We saw how we can use pass-in dependencies via function parameters:

We have the EmailScanner module that relies on a SpamFilterService to check if an email is spam or not:

defmodule EmailScanner do
  def scan_email(spam_filter_service, email) do
    spam_filter_service.check_spam(email)
  end
end

We have the SpamFilterService module that implements the spam checking logic:

defmodule SpamFilterService do
  def check_spam(email_content) do
    String.contains?(email_content, "spam")
  end
end

We also have a MockSpamFilterService module that implements the SpamFilterService behaviour for testing purposes:

defmodule MockSpamFilterService do
  def check_spam(_email), do: false
end

Finally, we have a test that uses the MockSpamFilterService to test the EmailScanner module:

defmodule EmailScannerTest do
  use ExUnit.Case

  test "scan_email with non-spam email returns false" do
    non_spam_email = %Email{content: "Hello, world!"}
    assert false == EmailScanner.scan_email(MockSpamFilterService, non_spam_email)
  end
end

In Elixir, modules are stateless, so the primary way to pass dependencies to a module is via function parameters. While Elixir modules can have attributes, these are used for compile-time information and metadata, not for holding runtime state.

Take the EmailScanner module, for example. We have to pass the SpamFilterService as a parameter to the scan_email function. This is unnecessary, as the only reason to have this function parameter is to make the module testable.

Additionally, it creates a few problems with code readability and navigation:

Because the module is expecting SpamFilterService as a parameter, we can't easily see what the module depends on.
The compiler can't catch issues with the module implementation, because we can pass any module that implements the SpamFilterService behaviour.

This approach might work well for small projects, but as our project grows, we might find ourselves repeating the same pattern over and over again. With Rewire, we don't have to worry about these issues. We can just focus on writing clean and maintainable code while keeping any testing concerns, mocks, and stubs in our test files.

Getting Started with Rewire and Mox in Your Elixir Project

Let's now dive into using Rewire and Mox in practice.

Step 1: Create a New Elixir Project

Before incorporating Rewire and Mox, create a new Elixir project:

mix new email_scanner

This command generates a new Elixir project named email_scanner, including a supervision tree structure.

Step 2: Add Dependencies

To use Rewire and Mox, you need to add them to your project's dependencies. Update your mix.exs file as follows:

defp deps do
  [
    {:rewire, "~> 1.0", only: :test},
    {:mox, "~> 1.0", only: :test}
  ]
end

After updating the dependencies, run mix deps.get in your terminal to fetch and install them.

Next, let's define our two primary modules:

defmodule EmailScanner do
  def filter_email(email) do
    email
    |> mark_as_important()
    |> SpamFilterService.check_spam()
  end

  defp mark_as_important(email) do
    important_senders = ["boss@example.com", "hr@example.com"]

    updated_email =
      if Enum.any?(important_senders, fn sender -> sender == email.sender end) do
        %{email | important: true}
      else
        email
      end

    updated_email
  end
end

We are making our code example a bit more realistic. The filter_email function marks emails from important senders as important and checks if the email is spam using the SpamFilterService module.

Next, we'll define the SpamFilterService module:

defmodule SpamFilterService do
  def check_spam(email_content) do
    String.contains?(email_content, "spam")
  end
end

Let's create a basic test for the EmailScanner module:

defmodule EmailScannerTest do
  use ExUnit.Case

  describe "filter_email/2" do
    test "marks email as important from specific sender and checks for spam" do
      important_sender_email = %{sender: "boss@example.com", content: "Please review the attached report.", important: false}
      non_important_sender_email = %{sender: "random@example.com", content: "Check out these deals!", important: false}

      # Filtering emails sent from the important sender
      assert %{important: true, is_spam: false} = EmailScanner.filter_email(important_sender_email)

      # Filtering emails sent from a non-important sender
      assert %{important: false, is_spam: false} = EmailScanner.filter_email(non_important_sender_email)
    end
  end
end

In the above code, the EmailScanner module relies on the SpamFilterService module to check if an email is spam or not. However, we can't test the EmailScanner module without also testing the SpamFilterService module, which is not ideal.

We need to mock the SpamFilterService module so that we can test the EmailScanner module in isolation.

Step 3: Configuring Mox

Mox requires a bit of setup in your test configuration. Open or create a test/test_helper.exs file and add the following line to define a mock based on a protocol or behaviour your project uses:

ExUnit.start()
Mox.defmock(EmailScanner.SpamFilterServiceMock, for: EmailScanner.SpamFilterService)

Mox makes it easy for us to generate mocks based on behaviours or protocols, which is essential for testing modules that rely on these abstractions.

Once our mock is defined, we can use it in our tests instead of the real implementation. With Rewire, we can inject these mocks into our modules without relying on function parameters.

Core Concepts of Rewire

Rewire simplifies the DI process in Elixir by providing a macro-based approach to define and inject dependencies. It fits seamlessly within Elixir’s ecosystem, promoting clean and maintainable code.

Dependency Injection with Rewire in the `EmailScanner` Module

Let’s implement the EmailScanner module, which relies on a SpamFilterService to check if an email is spam or not. Using Rewire, we can easily inject this dependency. Take a look at the following code:

defmodule EmailScanner do
  def filter_email(email) do
    email
    |> mark_as_important()
    |> SpamFilterService.check_spam()
  end

  defp mark_as_important(email) do
    important_senders = ["boss@example.com", "hr@example.com"]

    updated_email =
      if Enum.any?(important_senders, fn sender -> sender == email.sender end) do
        %{email | important: true}
      else
        email
      end

    updated_email
  end
end

Mocking with Mox for Testing

To test the EmailScanner filter function, we can use Mox to mock the SpamFilterService module:

defmodule EmailScannerTest do
  use ExUnit.Case

  import Rewire
  import Mox

  rewire EmailScanner, SpamFilterService: SpamFilterServiceMock

  # Ensure mocks are verified after each test
  setup :verify_on_exit!

  describe "filter_email/2" do
    test "marks email as important from specific sender and checks for spam" do
      important_sender_email = %{sender: "boss@example.com", content: "Please review the attached report.", important: false}
      non_important_sender_email = %{sender: "random@example.com", content: "Check out these deals!", important: false}

      # Stub the SpamFilter service to return false for all emails
      stub(SpamFilterServiceMock, :check_spam, fn _email -> :false end)

      # Filtering emails sent from the important sender
      assert %{important: true, is_spam: false} = EmailScanner.filter_email(important_sender_email)

      # Filtering emails sent from a non-important sender
      assert %{important: false, is_spam: false} = EmailScanner.filter_email(non_important_sender_email)
    end
  end
end

Let's break down what is happening in the above test:

rewire EmailScanner, SpamFilterService: SpamFilterServiceMock: This line uses Rewire to replace the SpamFilterService dependency in the EmailScanner module with SpamFilterServiceMock for the scope of this test module. It effectively changes the behavior of EmailScanner to use the mock service instead of its real dependency.
setup :verify_on_exit!: A setup callback that ensures all expectations on mocks (defined using Mox) are met by the end of each test, or else the test fails. This is crucial for verifying that the mocked functions are called as expected.
Then, we define a test case that:
- Creates two email maps, one from an "important" sender and one from a "non-important" sender.
- Uses stub to define the behavior of the SpamFilterServiceMock, so check_spam/1 always returns false, simulating a scenario where no email is considered spam.
- Calls filter_email/2 on both emails, expecting the function to correctly identify and mark the important email and to correctly interact with the spam filter (mocked to always return false for spam checks).

Under the hood, Rewire is doing a couple of interesting things. First, it's important to understand the philosophy behind Rewire and the approach the author decided to take. rewire works by using macros to create a copy of the module. So, for every test, Rewire creates a new module with the specified stubs.

Creating a copy of each module instead of overriding the original module allows us to run tests in parallel without any side effects.

Things to Consider When Using Rewire and Mox

When using Rewire and Mox in your Elixir projects, consider the following:

Asynchronous Testing Compatibility: Rewire fully supports asynchronous testing with async: true. Unlike global overrides used by tools like Meck, Rewire creates a separate module copy for each test. This ensures that tests can run in parallel without interfering with each other.
Integration with Mox: Rewire complements Mox perfectly by focusing on dependency injection without dictating the source of the mock module. This synergy allows for efficient and seamless integration between the two, making them an excellent pair for Elixir testing.
Impact on Test Speed: Rewire might slightly slow down your tests, although the effect is typically minimal. Comprehensive performance data from large codebases is still pending.
Test Coverage Accuracy: Yes, test coverage is accurately reported with Rewire, ensuring that you can trust your test coverage metrics.
Compatibility with Stateful Processes: Rewire works well with stateful processes, provided that these processes are started after their module has been rewired. For processes started beforehand (like a Phoenix controller), Rewire may not be effective since rewiring can no longer be applied. It's recommended to use Rewire primarily for unit tests where this limitation doesn't apply.
Erlang Module Rewiring: Rewire cannot directly rewire Erlang modules. However, it allows for Erlang module references to be replaced within Elixir modules, offering a workaround for this limitation.
Handling Nested Modules: Rewire will only replace dependencies within the specifically rewired module. Surrounding or nested modules will remain unaffected, maintaining references to the original modules. For complete control, you may need to rewire these modules individually.
Formatter Configuration for Rewire: To prevent mix format from adding parentheses around Rewire, update your .formatter.exs file with import_deps: [:rewire]. This ensures that Rewire syntax is correctly formatted without unnecessary parentheses.

And that's it!

Wrapping Up

In this post, we've explored how Rewire and Mox can help with dependency injection in Elixir.

Stephan Behnke, the creator of Rewire, was motivated by a desire for a more elegant solution to dependency injection in Elixir, especially for unit testing. I believe he succeeded in providing a great tool for the Elixir community.

That said, Rewire is not a silver bullet and it might not be the right tool for every project. It is important to evaluate Rewire alongside tools like Meck and make a decision based on your project and team's needs.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Using Dependency Injection in Elixir

Roy Tomeij — 2024-05-21T00:00:00+00:00

While controversial in functional programming, dependency injection can be a useful pattern in Elixir for managing dependencies and improving testability.

In this, the first part of a two-part series, we will cover the basic concepts, core principles, and types of dependency injection. We'll explore its benefits in terms of modularity, testability, and maintainability.

Then, we will look into a specific scenario where dependency injection can be beneficial, in this case, testing.

Let's first explain what dependency injection is.

What Is Dependency Injection?

Dependency Injection (DI) is a software design pattern that involves supplying an external dependency to a component rather than allowing the component to create the dependency itself. This pattern is a form of Inversion of Control (IoC), where control over the dependencies is inverted from the component to an external entity.

The main goal of DI is to reduce coupling between components, making our system more modular, flexible to changes, and easier to test.

These are the core concepts of dependency injection:

Dependency: An entity that another entity depends on to function properly.
Injector: The mechanism that injects dependencies into a component.
Client: The component that depends on the provided dependencies to operate.
Service: The dependency that the client component uses.

Types of Dependency Injection

Constructor Injection: The dependencies are provided through the component's constructor.
Setter Injection: The dependencies are provided through setter methods or properties.
Interface Injection: The dependency provides an injector method that will inject the dependency into any client passed to it.

Advantages of Dependency Injection

Reduced Coupling: By decoupling components from their dependencies, systems become more modular, allowing for easier maintenance and scalability.
Increased Flexibility: Changing or updating dependencies does not require changes to the dependent components, making the system more adaptable.
Improved Testability: Dependencies can be easily mocked or stubbed in tests, allowing for more isolated and reliable testing.

How Dependency Injection Works

There are four steps you should take to leverage dependency injection in your program or service:

Define the Service Interfaces: These interfaces represent the abstract contracts that services must fulfill.
Implement the Services: Concrete implementations of the service interfaces are developed.
Configure the Injector: The injector is configured to know which service implementations to inject into which clients.
Inject Dependencies: When a client is instantiated, the injector supplies it with the required service implementations based on the configuration.

Dependency Injection Diagram

The following diagram illustrates the basic concept of dependency injection:

The Client requires a service interface to perform its function.
The Dependency Injector decides which implementation of the service interface (Service A or Service B) to inject into the client at runtime.
Service A and Service B are different implementations of the same service interface. The injector injects one of these into the client based on the configuration or conditions.

This pattern allows for high flexibility and decoupling of components within software applications, facilitating easier management, testing, and evolution of the application code.

How Can Dependency Injection Be Applied in Elixir?

As we mentioned earlier, dependency injection is a pattern that is more commonly associated with object-oriented programming languages. Functional programming languages like Elixir offer a different set of tools and idioms for managing dependencies and state. However, the principles of DI can still be applied in Elixir to achieve similar benefits.

In Elixir, the emphasis on explicit over implicit dependency management aligns well with DI principles. For testing purposes, DI allows developers to easily replace real implementations with mocks or stubs, facilitating isolated unit tests that are not dependent on external services or state. This approach enhances test reliability and execution speed, as tests become less brittle and more focused on the functionality being tested.

Practical Application of Dependency Injection in Elixir for Testing

let's look at how we can use dependency injection to inject mocks and configure dependencies in Elixir.

Injecting Mocks

One common application of DI in Elixir testing involves injecting mock modules or functions that simulate the behavior of real dependencies. This technique is particularly useful when dealing with external services like databases or APIs.

defmodule MyApp.MyModule do
  def fetch_data(dataSource) do
    dataSource.query()
  end
end

defmodule MyApp.MyModuleTest do
  use ExUnit.Case

  test "fetch_data returns expected result" do
    mockDataSource = %{
      query: fn -> {:ok, "mocked data"} end
    }

    assert MyApp.MyModule.fetch_data(mockDataSource) == {:ok, "mocked data"}
  end
end

In this example, MyApp.MyModule.fetch_data/1 depends on a dataSource that responds to a query function. During tests, a mock dataSource is injected, allowing the test to run independently of any external data sources.

Configurable Dependencies

Another DI strategy involves using application configuration to define dependencies, which can then be overridden in the test environment.

# config/config.exs
config :my_app, data_service: MyApp.DataService

# config/test.exs
config :my_app, data_service: MyApp.MockDataService

In your application code, you would fetch the dependency from the application configuration:

defmodule MyApp.MyModule do
  def fetch_data do
    dataSource = Application.get_env(:my_app, :data_service)
    dataSource.query()
  end
end

This simple example shows how DI can be achieved in Elixir by configuring dependencies at runtime, allowing for easy substitution of real implementations with mocks or stubs during testing.

Next, let's review a more practical example that uses DI to inject a mock service into a module for testing.

Testing with Dependency Injection

In this example, we will work on EmailScanner, a module that scans emails for spam. We will use a SpamFilterService to check if emails are spam and dependency injection to inject a mock SpamFilterService for testing.

Start by creating a new Elixir project:

mix new email_scanner

Now let's move on to implementation and testing.

Implementation with `ExUnit`

First, create the EmailScanner module. This module will depend on a SpamFilterService to check if emails are spam. In this case, the SpamFilterService will be injected as a dependency, making it easy to swap with a mock during testing.

defmodule EmailScanner do
  def scan_email(spam_filter_service, email) do
    spam_filter_service.check_spam(email)
  end
end

Testing `EmailScanner` with `ExUnit`

Now, let's write a test for the EmailScanner module using ExUnit. We'll create a mock SpamFilterService to inject during tests:

defmodule MockSpamFilterService do
  def check_spam(_email), do: false
end

In this mock, the check_spam/1 function always returns false, simulating a non-spam email. Next, let's create a test case that makes use of our new mock:

defmodule EmailScannerTest do
  use ExUnit.Case

  test "scan_email with non-spam email returns false" do
    non_spam_email = %Email{content: "Hello, world!"}
    assert false == EmailScanner.scan_email(MockSpamFilterService, non_spam_email)
  end
end

This test injects MockSpamFilterService into EmailScanner, isolating the test from the real spam filtering logic and focusing solely on the EmailScanner's behavior.

By doing this, we can decouple the EmailScanner module from the SpamFilterService, making it easier to test and maintain.

Now that we've taken a look at using ExUnit and testing, let's turn to some common dependency injection mistakes to avoid and best practices.

Common Dependency Injection Pitfalls and Best Practices

First, we'll touch on some pitfalls:

Over-Reliance on Mocks: While DI makes it easy to replace real implementations with mocks, overusing mocks can lead to fragile tests that are overly focused on implementation details rather than behavior.
Complex Dependency Graphs: Introducing DI without careful planning can lead to a tangled web of dependencies that are hard to manage and understand.
Ignoring the Complexity of Configuration: DI often requires some form of configuration to wire up dependencies. This configuration can become complex and unwieldy if not managed properly.

To help you avoid these pitfalls, here are some best practices to follow:

Define Clear Interfaces: Ensure that your dependencies have clearly defined interfaces.
Use Configuration Wisely: Be mindful of the complexity that configuration can introduce.
Leverage Elixir’s Capabilities: Take advantage of Elixir’s features, such as module attributes and configuration files, to manage your dependencies effectively.
Test with Real Implementations When Possible: While mocking is useful, also test with real implementations to ensure that your system works as expected in real-world scenarios.

That's it for this part of the series!

Wrapping Up and What's Next

In this article, we have covered the basic concepts of dependency injection, its application in Elixir, and how it can be leveraged for testing. We have also discussed common pitfalls to avoid and best practices to follow when implementing DI in Elixir.

In the next and final part of this series, we'll look specifically at advanced dependency injection in Elixir using Rewire.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Deep Diving Into the Erlang Scheduler

Roy Tomeij — 2024-04-23T00:00:00+00:00

Erlang is renowned for its remarkable fault tolerance and high concurrency. Erlang's scheduler efficiently handles many lightweight processes. The scheduler plays a crucial role in managing processes, concurrency, and system resources, efficiently coordinating these elements to help Erlang maintain fault tolerance and support high levels of concurrency in its applications.

This post will dissect some of the scheduler's key components and shed light on how it works internally.

Let's get started!

Processes in Erlang

Erlang processes are lightweight, independent units of execution managed by the Erlang runtime system. They are created and scheduled by the Erlang virtual machine (BEAM), and each Erlang process has its own memory space. These should not be confused with operating system processes or threads.

OS processes are entities managed by the OS and typically have more overhead in terms of memory and resources, as well as inter-process communication. While threads are lighter than OS processes, they are still heavier than a typical Erlang process and share a common memory space within a process. Communication is usually easier between threads of the same process but still requires synchronization mechanisms.

On the other hand, the Erlang VM (BEAM) is capable of spawning several lightweight processes with independent memory space and can communicate easily using message passing. The scheduler inside the VM manages processes, allocates resources to processes, and context-switches between them.

Erlang Scheduler Architecture — Preemptive Scheduling

In a single-core setup, only one process can occupy the CPU core at any given time. To emulate a concurrent system, the scheduler employs a preemptive, priority-based scheduling mechanism (don't worry, we will explore what this means soon) that rapidly switches between available processes to create the illusion that all processes are executing simultaneously. The Erlang Virtual Machine (BEAM) schedules processes to run sequentially, with one process running for a duration, being suspended, and then allowing another process to take its turn. This process management strategy is characteristic of concurrency and does not entail true parallelism (which is not possible on a single-core system). Tasks appear to run concurrently due to fast context switching, although they actually execute sequentially on a single core.

To identify processes for potential swapping, Erlang introduces the concept of "reductions". In Erlang, a reduction represents a unit of work performed by BEAM, encompassing fundamental operations such as function application, arithmetic calculations, or message passing. The scheduler keeps track of the reductions executed by each process, preempting a process when it reaches a certain reduction count, thereby allowing another process to run.

Reductions

The idea of "reduction" in Erlang is inherited from its Prolog ancestry. In Prolog, every execution step is termed a goal-reduction, involving breaking down a logic problem into individual components and solving each part accordingly.

To promote fairness among processes, Erlang's preemptive scheduling relies on reductions rather than time slices. If a process exhausts its allocated reductions, it can be preempted, even if its execution isn't complete. This approach prevents a single process from monopolizing the CPU for an extended period, fostering fairness among concurrent processes. By using reductions as the foundation for preemption, Erlang mitigates the risk of processes starving for CPU time. This design ensures that every process, irrespective of its workload, is periodically allowed to execute.

Moreover, reductions are flexibly applied based on the type of operation a process is performing. For example, certain operations, such as I/O operations, may consume various reductions. This adaptability allows the scheduler to effectively handle different types of operations.

In other languages, traditional blocking I/O operations can lead to inefficient resource utilization, as threads might be blocked while waiting for I/O to complete. Erlang's asynchronous and non-blocking I/O model allows processes to continue executing other tasks while waiting for I/O operations to complete. This minimizes the impact of blocking operations on overall system performance.

Priority

Each process in Erlang can have a priority value that decides how often the scheduler will execute that process. A process priority can be set using the Process.flag/2 (or process_flag/2 on Erlang) function call:

iex> Process.spawn(
  fn ->
    Process.flag(:priority, :high)
    # ...
  end,
  [:link]
)

There are 4 priority levels: low, normal, high, and max. max is reserved for internal use in Erlang and should not be used in application code. Processes on each priority level get a separate run queue and are scheduled in the normal round-robin fashion as described above.

Except low and normal, Erlang executes the processes of each priority queue exclusively. This means that if there is a process in the max priority queue, only max priority processes will be executed, and all other processes will be blocked. Similarly, if there is a process in the high priority queue, low and normal processes will not be executed until all processes from the high priority queue are executed (or are non-runnable). low and normal queues behave slightly differently — the processes inside both queues are interleaved such that normal priority processes are executed more often than low priority ones, but normal processes do not block the execution of low priority processes.

Due to the blocking behavior of high priority processes, they should be used very rarely and only for short-lived tasks. Overusing high priority processes is bound to lead to outages and affect the responsiveness of an application, as all other regular OTP processes run on normal priority.

Another point that's important here is that Erlang places no restrictions on communication between different priority levels of processes. So a high-priority process can wait for a message from a lower-priority process. This is allowed, but will effectively lower the priority of the high-priority process.

Running on Multiple Cores

Up to this point, we've explored how the scheduler orchestrates processes within a single-core system. However, in a multi-core environment, where additional resources are available for parallel processing, the Erlang VM creates a dedicated "run queue" for each available core. This enables true parallelism, as multiple processes can run simultaneously (one on each available core). Within each run queue, all processes adhere to the preemptive scheduling mechanism we discussed earlier.

Typically, Erlang processes share the same run queue as their parent process, and a work-stealing algorithm may come into play to ensure load balancing. For instance, if there are two cores in the system and one consistently handles busy processes while the other remains relatively idle, the Erlang schedulers on both cores engage in periodic communication. This communication facilitates the movement of processes from the heavily loaded core to the less busy one, ensuring a more equitable distribution of workloads across both cores.

In the broader context, beyond Erlang's internal run queues, the operating system plays a role in managing the scheduling of threads onto OS-level cores. This implies that processes not only experience swapping within the Erlang run queue but also may undergo a complete context switch or be moved to a different core at the OS level.

Note that, because of the concept of work-stealing inside the Erlang VM, it is usually beneficial to run a single Erlang application instance on multiple cores rather than running separate instances of the same application on different cores of the same machine. In a single instance, the schedulers dedicated to each core can better communicate between them to share the load equitably compared to a multi-node cluster where schedulers cannot share process load (even if all of that cluster's nodes are on the same physical machine).

Performance and Optimization

Erlang's scheduler takes out most of the complexities involved in building a concurrent system. It automatically frees the developer from having to think about things like lock contention, thread overhead, and load balancing by handling these issues out of the box with its preemptive scheduling algorithm.

Erlang provides various scheduler-related parameters that can be tuned for optimal performance. Parameters like +S (scheduler count) and +P (maximum number of processes) allow you to configure the number of scheduler threads and processes.

For example, you can start Erlang with erl +S Schedulers:SchedulerOnline to control the number of scheduler threads. By default, Erlang uses the number of CPU cores to identify these values automatically. Note that while both Scheduler and SchedulerOnline accept values up to 1024, starting more schedulers than the number of CPU cores does not have any positive benefits for an app.

Another possible step to fine-tune performance is to control the priorities of processes, as we've discussed. It is indeed possible to execute certain high-priority tasks in Erlang.

Nevertheless, this comes with an inherent risk of potentially rendering a system unresponsive and increasing latency, as processes with a high priority may block all other normal/low-priority processes. Conversely, marking tasks identified as intentionally low priority can be advantageous to prioritize other processes above them.

So be careful and use your judgment.

Wrapping Up

In this post, we've seen that the Erlang scheduler stands as a cornerstone in Erlang's architecture, fostering fault tolerance, concurrency, and adaptability. Its preemptive and dynamic nature equips developers to build resilient, highly concurrent systems capable of handling failures and utilizing resources optimally. Understanding the intricacies of the Erlang scheduler can empower you to craft scalable and robust distributed applications.

If you are interested in learning more about the scheduler, I recommend checking out the Scheduling chapter in The BEAM Book and Processes from the Erlang Efficiency Guide.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

P.P.S. AppSignal has an integration for Erlang — check it out.

How to Use Flume in your Elixir Application

Roy Tomeij — 2024-04-02T00:00:00+00:00

As your Elixir app grows, you might need advanced control over how and where to perform background tasks or pull them off queues to manage back pressure.

In this post, you will learn how to handle background jobs with Flume, a job processing system that uses GenStage and Redis. It provides durability, back pressure, job scheduling, rate limiting, and batch processing, among other things.

We will expand on each of these features in detail.

But before we go further, let's quickly cover two other major libraries in the same space: Oban and Exq.

Oban and Exq: Flume Alternatives for your Elixir App

Both Oban and Exq serve as excellent alternatives to Flume.

Oban, backed by PostgreSQL or SQLite, also provides a queue-based job processing system. Exq, on the other hand, is backed by Redis. It provides features similar to Flume, but without built-in rate limiting and batch processing capabilities.

Opt for Oban if you don't want to maintain a Redis instance to manage background jobs (Postgres typically performs adequately unless faced with queues exceeding 100k jobs or more) and do not require rate limiting.

Select Exq when you need a highly efficient queuing backend for processing a substantial volume of jobs, but can forgo advanced features such as rate limiting and batch processing.

If you need all three features (a queuing backend, rate limiting, and batch processing), Flume emerges as the ideal choice.

A small word of warning: If you have experience with Rails, you are likely familiar with Sidekiq, Resque, and Delayed Job for background job processing. But note that Elixir/Erlang already has excellent alternatives built into the language in the form of GenServer, Task, and Supervisor, and an additional background job processing system that adds more infrastructure costs and complexity might not be necessary.

A background job processing system like Flume becomes crucial when you need to manage numerous background tasks, track their progress/status (or at least automatically retry them on failure), or apply common constraints (like queue, priority, or rate limit) to many similar jobs.

Keeping all of that in mind, let's set up Flume for our Elixir app.

Installation and Setting Up

To install Flume in your app, simply add the dependency in your mix.exs file and run mix deps.get:

def deps do
  [
    {:flume, github: "scripbox/flume"}
  ]
end

Next, add Flume to your application’s supervision tree by modifying your application.ex file:

defmodule MyApp.Application do
  use Application

  import Supervisor.Spec

  @impl true
  def start(_type, _args) do
    Supervisor.start_link([
      # ...
      # Start Flume supervisor
      supervisor(Flume, []),
      #...
    ], strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

Finally, configure it inside config/config.exs:

config :flume,
  name: Flume,
  # Redis config
  host: "127.0.0.1",
  port: "6379",
  namespace: "my-app",
  database: 0,
  redis_pool_size: 10

There are some other configuration options as well, but we will get to them later in the post. For now, this is all you need to get things set up and start creating background jobs. Let's create one now:

defmodule MyApp.OrdersWorker do
  def process(order_id) do
    # process the order
  end
end

To initiate a job, we execute Flume.enqueue/4, specifying a queue name, a module name, a function within that module, and an array of arguments to invoke that function:

Flume.enqueue(:default, MyApp.OrdersWorker, :process, [order_id])

This takes care of serializing the arguments and adds the job to Redis. To actually run the job, we need to define a pipeline that picks off the jobs from Redis. We'll see how to do that in the next section.

Pipelines in Flume for Elixir

Pipelines define when and how the actual job processing happens in Flume. We can define pipelines in config/config.exs when configuring Flume:

config :flume, pipelines: [%{name: "default_pipeline", queue: "default"}]

This directs Flume to commence processing jobs from the queue named default, with a maximum of 500 jobs (by default) running simultaneously. Consequently, Flume will initiate up to 500 concurrent processes/tasks, each executing a single job. The max_demand parameter regulates this behavior. If your jobs are inherently resource-intensive, you can specify a lower number to constrain the quantity of workers:

config :flume, pipelines: [%{name: "default_pipeline", queue: "default", max_demand: 10}]

In addition to the queue, there are more configuration options for advanced pipeline control.

Rate Limiting

We can specify rate limits for a pipeline by configuring it like this:

config :flume, pipelines: [%{
  name: "default_pipeline",
  queue: "default",
  rate_limit_count: 100,
  rate_limit_scale: 60 * 1000
}]

This will ensure that the default pipeline processes at most 100 jobs (rate limit count) per minute (rate limit scale of 60 * 1000 milliseconds). It is also possible to share the same rate limit across multiple pipelines by specifying the optional rate_limit_key when defining the pipeline.

For example, if you have two pipelines that process notifications using the same rate-limited API, you can share the limit across both of them like this:

config :flume, pipelines: [
  %{
    name: "email_pipeline",
    queue: "email",
    rate_limit_count: 100,
    rate_limit_scale: 60 * 1000
    rate_limit_key: "notifications"
  },
  %{
    name: "sms_pipeline",
    queue: "sms",
    rate_limit_count: 100,
    rate_limit_scale: 60 * 1000
    rate_limit_key: "notifications"
  }
]

Now, both of these pipelines will process a total of (at most) 100 jobs per minute.

Don't confuse this with the max_demand parameter, which governs how many "concurrent" jobs can execute simultaneously. Even with a low max demand, it's entirely possible to rapidly process too many jobs if they tend to be short-lived. Rate limiting operates on a time scale level to ensure that the number of executions does not surpass a specified limit within a given time interval.

Batching

It's also useful to define a batch size with a pipeline:

config :flume, pipelines: [%{
  name: "default_pipeline",
  queue: "default",
  max_demand: 10,
  batch_size: 5,
}]

This will instruct Flume to send up to 5 jobs to the same worker (and start up to 10 workers at the same time).

When using batching, each worker will receive an array of all arguments:

defmodule MyApp.OrdersWorker do
  def process(jobs) do
    # jobs contains up to 5 arrays containing arguments for each individual call:
    # [[order1_id], [order2_id], [order3_id], [order4_id], [order5_id]]
  end
end

The application code determines how to manage these jobs. For instance, in the case of a mailer job, the process/1 function could utilize a bulk email API to send emails to multiple recipients in a single call, rather than making five separate calls to a standard API.

Runtime Control

It is also possible to pause pipelines at runtime using Flume.pause_all/1 or Flume.pause/2.

To pause all pipelines permanently (across all running nodes), run:

Flume.pause_all(temporary: false, async: true)

If you only want to pause a single pipeline, use Flume.pause with the pipeline's name. It accepts the same options as pause_all:

Flume.pause("default_pipeline", temporary: false, async: true)

It is also possible to pause a pipeline/all pipelines on a single node instead of on all nodes. This can be controlled using the temporary parameter. So, to pause a pipeline on only the current node, run:

Flume.pause("default_pipeline", temporary: true, async: true)

Similar to pause and pause_all, you can run resume or resume_all to restart the pipelines.

Pausing a pipeline can be advantageous in scenarios where you need to temporarily suspend job processing to address issues or perform maintenance tasks. For example, during system upgrades or when debugging critical issues, pausing a pipeline can prevent new jobs from being processed, thereby enabling you to stabilize the system.

Another useful scenario is to manage known service downtimes. For instance, if your job depends on an external service that is known to be unavailable, you can pause the pipeline until the service resumes normal operation.

Advanced Features in Flume

Flume comes with an exponential back-off for failed jobs (jobs that raise an exception) by default. The default configuration is set to retry a failed job with exponential back-offs starting at 500 milliseconds with a maximum back-off of 10 seconds. A job is retried at most 5 times. This can be controlled using:

config :flume,
  backoff_initial: 30_000,
  backoff_max: 36_00_000,
  max_retries: 10

There are several other configuration options to fine-tune Flume further. Please refer to the Flume guide for more details.

Flume uses :telemetry to emit events/metrics. By default, the emitted events are:

A [:pipeline_name, :worker, :job] event with the duration (in milliseconds) representing the time taken to execute the job. This is the time that your Worker.perform function takes.
A [:pipeline_name, :worker] event with the duration (in milliseconds) representing the time taken to run the job. This includes the total time taken to decode the event payload and execute the job.
A [:queue_name, :enqueue] event with the serialized job's payload_size (in bytes).
A [:queue_name, :dequeue] event with the count (number of jobs fetched), latency (in milliseconds — time taken to fetch the jobs from Redis), and payload_size (in bytes) after Flume fetches jobs from Redis.

If you are already using AppSignal, you can use something like TelemetryMetricsAppsignal to collect these metrics, report them to AppSignal, and get custom dashboards.

Running Flume On Production

All the usage examples we have seen up to now run Flume on the same server as the web server. While this is good for a development workflow or small production workloads, you might consider moving workers to a separate node for scalability. This can be achieved by configuring Flume pipelines based on environment variables.

To control this, we can create a module that helps parse the environment variable:

defmodule MyApp.FlumeConfigurator do
  @default_pipelines %{
    "default_pipeline" => %{name: "default_pipeline", queue: "default"},
    "notifications_pipeline" => %{
      name: "notifications_pipeline",
      queue: "notifications",
      rate_limit_count: 100,
      rate_limit_scale: 60 * 1000
    },
    "video_transcoding_pipeline" => %{name: "video_transcoding_pipeline", queue: "video_transcoding", max_demand: 5}
  }

  def flume_pipelines() do
    pipelines(System.get_env("FLUME_PIPELINES"))
  end

  defp pipelines(nil), do: []

  defp pipelines(value) when is_binary(value) do
    value
    |> String.split(",", trim: true)
    |> Enum.map(fn name -> Map.fetch!(@default_pipelines, name) end)
  end
end

And then configure Flume pipelines inside runtime.exs/release.exs:

import MyApp.FlumeConfigurator, only: [flume_pipelines: 0]
config :flume, pipelines: flume_pipelines()

That way, if you start your released application without defining FLUME_PIPELINES, Flume will not start any pipelines, but you can still enqueue jobs from your application. Then you can start other workers with FLUME_PIPELINES so that all pipelines kick in:

FLUME_PIPELINES=default_pipeline,notifications_pipeline,video_transcoding_pipeline /path/to/release start

You might want to stop the web server on worker-only nodes: you can control that from the runtime.exs/release.exs file with an environment variable.

Note that this is automatically generated if you use the mix phx.gen.release task to prepare your Phoenix app for release:

if System.get_env("PHX_SERVER") do
  config :pug_n_play_platform, PugNPlayPlatformWeb.Endpoint, server: true
end

Best Practices

When enqueueing background jobs with Flume, all the job arguments are serialized and deserialized, and all the job data needs to make a round trip to the Redis server. This means that it is important to keep the payload of the jobs as low as possible. For example, if you have something that persists in a database, instead of passing the full struct, just pass the ID and retrieve it from the database when performing the job.

Another important thing to keep in mind when running jobs is that they should be idempotent. Jobs can run multiple times in cases of failure, so ensure that a job that runs twice doesn't unnecessarily affect your application or its data (or disable retries).

Wrapping Up

In summary, Flume brings efficiency to background job processing in Elixir applications. Its easy installation, configurable pipelines, and advanced features like rate limiting and telemetry make it a valuable tool for scalable and fault-tolerant systems. Embracing best practices, such as minimizing payload and ensuring idempotency, enhances the effectiveness of your background jobs.

With Flume, you not only leverage Elixir's concurrency and fault tolerance but also gain a powerful ally for managing and optimizing your application's background tasks.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Direct File Uploads to Amazon S3 with Phoenix LiveView

Roy Tomeij — 2024-03-19T00:00:00+00:00

In this post, we'll add file upload capabilities to a Phoenix LiveView application and directly upload files to Amazon S3.

Without further ado, let's get started!

Setting Up Our Phoenix LiveView Project

Our example will upload behavior to an existing application with a “create puppy” feature. We're going to go through the Phoenix LiveView Uploads guide, making small adjustments as we go.

To begin, we need to set up our project. You can skip this step if you already have an existing Phoenix LiveView application. Otherwise, you can create a new project by running the following command in your terminal:

mix phx.new puppies

Change into the project directory:

cd puppies

And create a new Phoenix LiveView:

mix phx.gen.live Puppies Puppy puppies name:string breed:string color:string photo_url: string

This will generate the necessary files and migrations for our live view and model.

Allowing Uploads

The very first thing we need to do is enable an upload on mount. Open the form component file at lib/puppies_web/live/puppy_live/form_component.ex. You will see an update/2 function where we will enable file uploads. Add the following line to your callback:

allow_upload(socket, :photo, accept: ~w(.png .jpeg .jpg .webp), max_entries: 1, auto_upload: true)

With the update/2 callback ultimately looking like this:

@impl true
def update(%{puppy: puppy} = assigns, socket) do
  changeset = Puppies.change_puppy(puppy)

  {:ok,
   socket
   |> allow_upload(:photo, accept: ~w(.png .jpeg .jpg .webp), max_entries: 1, auto_upload: true, external: &presign_entry/2)
   |> assign(assigns)
   |> assign(:form, to_form(changeset)}
end

This line enables file uploads for the :photo attribute, accepting only JPEG, WEBP, and PNG file formats and only one upload at a time via max_entries: 1. With auto_upload: true, a photo begins uploading as soon as a user selects it in our form. You can modify these options based on your requirements by looking at the allow_upload function docs.

If you’re not using a LiveComponent, you can add this line to your mount/2 callback.

Adding the Upload Markup

Next, we’re going to render the HTML elements in our form. This is where we will put our file input, a preview of our picture, and the upload percentage status. We'll also add drag-and-drop support for images.

To render the file upload form in our LiveView, we need to add the necessary HTML components. In the puppy form HTML, add all of these file upload goodies:

<div>
  <%= hidden_input @form, :photo_url %> <%= error_tag @form, :photo_url %>

  <div phx-drop-target="{@uploads.photo.ref}">
    <.live_file_input upload={@uploads.photo} />
  </div>

  <div>
    <%= for {_ref, msg} <- @uploads.photo.errors do %>
    <h3><%= Phoenix.Naming.humanize(msg) %></h3>

    <% end %> <%= for entry <- @uploads.photo.entries do %> <.live_img_preview
    entry={entry} width="75" />
    <div class="py-5"><%= entry.progress %>%</div>
    <% end %>
  </div>
</div>

In this code, we render a file input with drag-and-drop support via the phx-drop-target form binding and live_file_input component provided by Phoenix LiveView. We also display a live file preview of the uploaded photo with <.live_img_preview entry={entry} width="75" />.

Live updates to the preview, progress, and errors will occur as the end-user interacts with the file input.

We render an image preview for each uploaded image stored in @uploads.photo.entries. The @uploads variable becomes available to us when we pipe into the allow_upload/3 function in our form component (remember, you can do this in the mount/2 callback for a LiveView if you aren’t using a LiveComponent).

for {_ref, msg} <- @uploads.photo.errors will display any errors, like uploading the wrong file type, to the user.

Finally, notice this:

<%= hidden_input @form, :photo_url %>
<%= error_tag @form, :photo_url %>

These appear in the form because after we upload the file to Amazon S3, we’re going to persist the url of the picture to the database.

Important: Your LiveView upload form must implement both phx-submit and phx-change callbacks, even if you’re not using them.

The LiveView upload feature set will not work without these callbacks and subsequent handle_event callbacks. Read more about this.

Configuring Amazon S3 to Phoenix LiveView

Before we can test our file upload feature, we need to set up our Amazon S3 bucket and configuration. Setting up your Amazon S3 bucket and configuring access is outside the scope of this guide, but this video should help.

Consuming the File Upload on the Back-end

We’re going to crack open the Direct to S3 guide to bring all of this home.

In our allow_upload function call, add “external” support so that we can get it on Amazon S3. It should look like this:

def update(%{puppy: puppy} = assigns, socket) do
  changeset = Puppies.change_puppy(puppy)

  {:ok,
   socket
   |> allow_upload(:photo, accept: ~w(.png .jpeg .jpg .webp), max_entries: 1, auto_upload: true, external: &presign_entry/2)
   |> assign(assigns)
   |> assign(:changeset, changeset)}
end

defp presign_entry(entry, %{assigns: %{uploads: uploads}} = socket) do
  {:ok, SimpleS3Upload.meta(entry, uploads), socket}
end

external: &presign_entry/2 sends the file to our presign_entry/2 function. This function dedicates work to a SimpleS3Upload module that generates a “presigned_url”. We need to upload files directly to Amazon S3 via our front-end safely, without exposing our Amazon S3 credentials.

So, we generate a pre-authenticated short-term URL on the back-end with our Amazon credentials. We then send this URL back to the front-end, where it can safely upload our file to Amazon S3, already completely safely authenticated.

You can find the content of the SimpleS3Upload file here. It’s 170+ lines of code purely dedicated to creating a pre-signed url for pre-authenticated file uploading.

Inside the SimpleS3Upload module, you’ll see references to the configuration you need to add.

defp config do
  %{
    region: region(),
    access_key_id: Application.fetch_env!(:puppies, :access_key_id),
    secret_access_key: Application.fetch_env!(:puppies, :secret_access_key)
  }
end

In your app, you’ll need to specify your secret keys/credentials in your config/config.exs.

config :puppies,
  access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
  secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
  bucket: System.fetch_env!("S3_BUCKET_NAME"),
  region: System.fetch_env!("AWS_REGION")

Now let's pull this through to the front-end!

Uploading the File to S3 on the Front-end

We now need to upload the file from our front-end JavaScript in assets/js/app.js:

let Uploaders = {}

Uploaders.S3 = function(entries, onViewError){
  entries.forEach(entry => {
    let formData = new FormData()
    let {url, fields} = entry.meta
    Object.entries(fields).forEach(([key, val]) => formData.append(key, val))
    formData.append("file", entry.file)
    let xhr = new XMLHttpRequest()
    onViewError(() => xhr.abort())
    xhr.onload = () => xhr.status === 204 ? entry.progress(100) : entry.error()
    xhr.onerror = () => entry.error()
    xhr.upload.addEventListener("progress", (event) => {
      if(event.lengthComputable){
        let percent = Math.round((event.loaded / event.total) * 100)
        if(percent < 100){ entry.progress(percent) }
      }
    })

    xhr.open("POST", url, true)
    xhr.send(formData)
  })
}

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: {_csrf_token: csrfToken}
})

This will upload each one of our files to Amazon S3 via AJAX.

Notice how we have an event listener for a “progress” event, always keeping the upload status available via entry.progress(percent). Remember, we’re displaying the progress percentage in our HTML. And if there’s an error in the AJAX response, entry.error() has us covered. We render any file upload errors back to the user in the HTML, too.

The URL it’s uploading to is the pre-signed one that we generated on the back-end.

Saving the Amazon URL to the Database

In our form, there's a form submit binding that looks like phx_submit: "save", meaning we have a matching handle_event in our LiveView/form_component.

Make the following change:

def handle_event("save", %{"puppy" => puppy_params}, socket) do
  puppy_params = put_photo_urls(socket, puppy_params)
  save_puppy(socket, socket.assigns.action, puppy_params)
end

defp put_photo_urls(socket, puppy) do
  uploaded_file_urls = consume_uploaded_entries(socket, :photo, fn _, entry ->
    {:ok, SimpleS3Upload.entry_url(entry)}
  end)

  %{puppy | "photo_url" => add_photo_url_to_params(List.first(uploaded_file_urls), puppy["photo_url"])}
end

defp add_photo_url_to_params(s3_url, photo_url) when is_nil(s3_url), do: photo_url
defp add_photo_url_to_params(s3_url, _photo_url), do: s3_url

We’re not adding the Amazon S3 photo URL to puppy_params. The consume_uploaded_entries function call is where this magic is happening.

SimpleS3Upload.entry_url(entry) is the dynamically generated URL where our image lives in our S3 bucket.

Notably, observe my pattern matching in the add_photo_url_to_params function. This guards us from overwriting the puppy with a blank URL if we don’t upload a file when updating the puppy’s attributes.

Here's the final result:

Wrapping Up

You now know how to upload from Phoenix LiveView directly to Amazon S3.

We began by enabling file uploads in our LiveView and adding the necessary components to our HTML template. We then created a pre-signed (pre-authenticated) URL to upload the file to with the SimpleS3Upload module. Next, we actually performed the file upload via AJAX calls by adding the uploader to our application’s JavaScript. Finally, we consumed the files we uploaded, persisting the URL to the database.

Have fun experimenting with this feature and enhancing your application!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Powerful Caching in Elixir with Cachex

Roy Tomeij — 2024-03-05T00:00:00+00:00

Developers often initially look to the Elixir language and stack because it's known for being able to handle massive amounts of concurrent requests and scale easily. This makes Elixir a great choice for building highly performant applications.

However, sometimes operations are computationally expensive and can slow down your application. This is where caching comes in.

In this article, we'll explore how Cachex, a powerful library tailored for Elixir, can help you add caching to your application and improve its performance.

Understanding Caching and Its Importance in Elixir

Caching is the art of storing data temporarily to reduce redundancy and improve access times. The primary benefits of caching include faster data retrieval, reduced load on primary data sources, and enhanced application responsiveness. However, caching is not without its drawbacks. Over-reliance can lead to stale data, and improper cache management can result in increased complexity.

Caching can be added to reduce bottlenecks and improve performance in Elixir: for example, when dealing with external systems, databases, or computationally expensive operations. Caching can also be used to store frequently accessed data in memory, such as user sessions or application state.

Now let's take a quick look at some benefits and disadvantages of caching.

Benefits of Caching

Caching comes with several benefits, including:

Improved performance: Caching can significantly reduce data retrieval times, making applications more responsive.
Reduced load on primary data sources: By serving data from a cache, there's less strain on primary data sources like databases, reducing the risk of them becoming a bottleneck.
Cost savings: Reducing the number of calls to external services or databases can lead to cost savings, especially if those calls are billable.
Enhanced user experience: Faster response times lead to a smoother user experience.
Scalability: Caching can help applications handle more users simultaneously by reducing the need for resource-intensive operations.
Reduced network traffic: Serving data from the cache can reduce the amount of data that needs to be transmitted over a network.
Offline access: In some scenarios, caching allows users to access certain pieces of data even when offline.

Drawbacks of Caching

Even though there are a lot of benefits to caching, there are some drawbacks you should be mindful of:

Stale data: Cached data can become outdated, leading to users receiving old or incorrect information.
Increased complexity: Implementing caching introduces another layer of complexity to system architecture and can complicate debugging.
Memory usage: Caching, especially when done extensively, can consume a significant amount of memory.
Cache invalidation: Deciding when and how to invalidate or refresh the cache can be challenging.
Cache warm-up: After a cache clear or system restart, the cache might be "cold" and it can take time to "warm up" to an optimal state.
Potential for cache thrashing: Rapidly adding and evicting items can lead to cache thrashing, where the cache doesn't provide its benefits effectively.
Maintenance overhead: Over time, your caching strategy might need adjustments, leading to additional maintenance work.

It's worth noting that while caching offers numerous advantages, it's essential to implement it judiciously, considering the specific needs and characteristics of each application.

Caching Options in Elixir

In the world of Elixir, many different options are available for caching. It is important to understand that each library has its own set of pros and cons. As developers, we need to consider the specific scenario where we intend to use caching and choose the right tool for the job.

Some of the options available to use are:

Cachex: A powerful caching library tailored for Elixir. It offers features like Time-to-Live (TTL), fallbacks, and locking mechanisms.
ConCache: A concurrent caching library for Elixir. It's built on top of Erlang Term Storage (ETS) and provides features like TTL and cache partitioning.
Nebulex: A flexible and highly configurable caching library. It supports different caching strategies and has built-in support for distributed caching.
Erlang Term Storage (ETS): While not exclusively a caching solution, ETS is an in-memory store that's often used for caching in Elixir applications. It's a core feature of the Erlang runtime system.
Mnesia: A distributed database management system that comes with Erlang/OTP. It can be used for caching, especially in distributed Elixir applications.
Redis via Redix: While Redis is not an Elixir-specific solution, it's a popular choice for caching in many applications. The Redix library allows Elixir applications to interact with Redis easily.
Least Recently Used (LRU) caches: There are several Elixir libraries, like lru_cache, that implement the LRU caching algorithm.

For the rest of this article, we'll focus on Cachex, because it's a robust caching solution that's easy to use and offers a wide range of features. It's also well-documented and has a vibrant community around it.

Introduction to Cachex for Elixir

Cachex offers a suite of features that make it an indispensable tool for Elixir developers. From simple key-value storage to advanced features like TTL settings and fallback mechanisms, Cachex provides a comprehensive caching solution.

Among the many features that make Cachex a powerful tool for caching in Elixir, the following stand out:

Performance: Cachex is designed for high performance, ensuring rapid data retrieval and insertion. This is crucial for applications that require real-time responsiveness.
Concurrency support: Elixir is known for its concurrency capabilities, and Cachex is built to handle concurrent operations seamlessly. It ensures that multiple processes can interact with the cache without causing data inconsistencies.
Advanced features, including TTL (allowing developers to specify how long an item should remain in the cache, to ensure that data doesn't become stale), fallbacks (mechanisms to compute values if they're missing from the cache, so that an application can still function even if a cache miss occurs), and locking mechanisms (ensuring data integrity during operations like cache updates).
Flexibility: Cachex is not just a simple key-value store. It supports complex data structures, making it versatile for a range of applications.
Distributed caching: While Cachex primarily operates as a local cache, it can be combined with other tools and libraries to support distributed caching scenarios, making it suitable for clustered Elixir applications.
Comprehensive documentation: Cachex comes with extensive documentation, making it easier for developers to get started and harness its full potential.
Integration with Telemetry: Cachex integrates with the Telemetry library, allowing developers to gather metrics and monitor cache performance in real-time.
Cache eviction strategies: Cachex supports various cache eviction strategies, such as LRU, ensuring that the cache remains efficient even as it fills up.
Fault tolerance: Built with Elixir's fault-tolerant nature in mind, failures are isolated and don't bring down the entire application.

Adding Caching to Your Elixir Application with Cachex

To get started with Cachex, we'll need to add it as a dependency to our project.

To install Cachex in an Elixir application, follow these steps:

Add cachex as a dependency: Open your mix.exs file and add :cachex to the list of dependencies:

defp deps do
 [
   {:cachex, "~> 3.3"} # The version number might change over time, so always check for the latest version.
   ]
end

Fetch and compile dependencies:

Run the following command in your terminal:

mix deps.get

This will fetch and compile the Cachex library along with any of its dependencies.

Start Cachex:

Before using Cachex in your application, you need to start it. You can start a new cache instance with the following command:

{:ok, _} = Cachex.start_link(:my_cache)

Here, :my_cache is the name of the cache. You can choose any name that suits your application.

Calling Cachex.start_link/1 will start a new cache instance with the default configuration. If you want to customize the configuration, you can pass in a map with the desired configuration options.

For example, if you want to set the cache size to 1000 items, you can do so by passing in the :size option:

{:ok, _} = Cachex.start_link(:my_cache, %{size: 1000})

It's important to note that Cachex is built on top of GenServer, so you can use the same techniques to start and supervise it.

Add to application supervision tree: If you want the cache to be supervised and automatically restart in case of failures, you can add it to your application's supervision tree. This ensures that the cache is always available throughout the lifecycle of your application.

In your application's supervisor, you can add:

children = [
 {Cachex, name: :my_cache}
]

This will start the Cachex cache when your application starts and supervise it.

Note: There are some scenarios where you might not want to start Cachex automatically. For example, if you're using Cachex in a Phoenix application, you might want to start it only when the application is running in production. In such cases, you can start Cachex manually in your application's supervision tree.

That's it! You've successfully installed Cachex in your Elixir application. You can now use its various functions to cache data, retrieve cached data, and manage your cache. Always refer to the official Cachex documentation for more detailed information and best practices.

Working with Phoenix

A common use case for Cachex is to cache data in a Phoenix application. Phoenix doesn't have a built-in caching mechanism, so developers often turn to third-party libraries like Cachex.

Before we can use Cachex in a Phoenix application, we need to add it as a dependency and initialize it. We can do this by following the steps outlined in the previous section.

Once Cachex is installed, we can use it in our Phoenix application. Let's look at a few examples of using Cachex in a Phoenix application.

Caching Database Queries

One common use case in Phoenix applications is to cache the results of database queries to reduce database load and speed up data retrieval. After fetching data from the database, you can store it in the Cachex cache:

def get_user(id) do
  case Cachex.get(:my_cache, id) do
    {:ok, nil} ->
      user = Repo.get(User, id)
      Cachex.put(:my_cache, id, user)
      user
    {:ok, user} -> user
  end
end

The function get_user/1 retrieves a user by their id. It first checks if the user is available in the cache, and if not, it fetches them from a repository (likely a database) and then caches the result for future calls. Let's break down what's happening here:

Function definition: The function get_user/1 is defined to accept a single argument, id, which identifies a user.
Cache lookup: The function starts by trying to retrieve the user from a cache named :my_cache using the provided id as the cache key. Cachex.get(:my_cache, id) attempts to get the value associated with the key id from :my_cache.
Handling cache results: The result of the cache lookup is pattern-matched using a case statement to determine the next steps.
- Cache miss {:ok, nil}: If the cache returns {:ok, nil}, it indicates a cache miss, meaning the user is not present in the cache.
  - The function then fetches the user from a repository using Repo.get(User, id). This is likely a call to a database to retrieve the user data.
  - Once the user data is fetched, it's stored in the cache using Cachex.put(:my_cache, id, user). This ensures that subsequent calls for the same user id will find the user in the cache and won't need to hit the database.
  - Finally, the fetched user data is returned as the result of the function.
- Cache hit {:ok, user}: If the cache returns {:ok, user} where the user is not nil, this is a cache hit. The user data has been found in the cache. The function simply returns the cached user data without making any database calls.

Caching Views

Sometimes, we might need to cache rendered views to improve performance. For example, if we have a view that's expensive to render, we can cache it to reduce the load on the server and speed up data retrieval.

There are many reasons why views might be expensive to render. For example, they might contain complex logic or require multiple database calls. In such cases, caching can be a useful technique to improve performance.

We can use a plug to cache views in Phoenix. Here's an example:

Create a plug that checks if the view is cached and sends the cached content if it is. Otherwise, it continues with the pipeline and caches the response later.

defmodule MyApp.CacheViewPlug do
  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _opts) do
    cache_key = "view_cache:#{conn.request_path}"

    case Cachex.get(:my_cache, cache_key) do
      {:ok, nil} ->
        # Cache miss, continue with the pipeline and cache the response later
        conn
      {:ok, cached_content} ->
        # Cache hit, send the cached content and halt the pipeline
        conn
        |> put_resp_content_type("text/html")
        |> send_resp(200, cached_content)
        |> halt()
    end
  end
end

Let's break down what's happening here:

Module definition: the defmodule MyApp.CacheViewPlug do: line defines a new module named MyApp.CacheViewPlug.
Importing Plug.Conn: import Plug.Conn: imports functions from the Plug.Conn module, which provides utilities for working with connection structs in the context of a Plug.
Initialization function: def init(opts), do: opts: is the initialization function required by the Plug behavior. It takes an opts argument (options) and simply returns it unchanged. In many Plugs, this function sets default options or validates the provided options. However, in this case, it's a simple pass-through.
Call function: def call(conn, _opts) do: is the main Plug function that gets executed for every request. It takes two arguments: conn (the connection struct) and _opts (the options, which are ignored in this case).
Generating the cache key: cache_key = "view_cache:#{conn.request_path}": constructs a cache key based on the request path. The cache key is prefixed with "view_cache:" to namespace or differentiate it from other potential cache keys.
Checking the cache: The case Cachex.get(:my_cache, cache_key) do statement checks the cache (:my_cache) for content associated with the generated cache_key.
Handling cache results:
- {:ok, nil} ->: This pattern matches a cache miss. If the cache doesn't have content for the given key, it returns {:ok, nil}. In the event of a cache miss, the function simply returns the unchanged conn, allowing the request to continue through the Plug pipeline. The intention is that the response will be cached later, presumably by another part of the application.
- {:ok, cached_content} ->: This pattern matches a cache hit. If the cache has content for the given key, it returns {:ok, cached_content}. In this case, the function sends the cached content as the response, sets the response content type to "text/html", and then halts the Plug pipeline to prevent further processing.

Now, add the plug to your pipeline in router.ex:

pipeline :browser do
  ...
  plug MyApp.CacheViewPlug
  ...
end

Finally, cache the view after rendering it:

def some_action(conn, _params) do
  content = render(conn, "some_template.html")
  cache_key = "view_cache:#{conn.request_path}"
  Cachex.put(:my_cache, cache_key, content)
  send_resp(conn, 200, content)
end

Caching API Responses

Another common Cachex use case for Phoenix is to cache API responses. This can be useful to reduce the load on external services and speed up data retrieval. Similar to caching views, we can use a plug to cache API responses in Phoenix.

Create a plug that checks if the response is cached and sends the cached content if it is. Otherwise, it continues with the pipeline and caches the response later.

defmodule MyApp.CacheAPIPlug do
  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _opts) do
    # Create a cache key based on the request path and query parameters
    cache_key = "api_cache:#{conn.request_path}?#{conn.query_string}"

    case Cachex.get(:my_cache, cache_key) do
      {:ok, nil} ->
        # Cache miss, continue with the pipeline and cache the response later
        conn
      {:ok, {status, headers, body}} ->
        # Cache hit, send the cached response and halt the pipeline
        conn
        |> put_status(status)
        |> put_resp_header("content-type", List.keyfind(headers, "content-type", 0, {"content-type", "application/json"}))
        |> send_resp(status, body)
        |> halt()
    end
  end
end

Add the plug to your pipeline in router.ex:

pipeline :api do
  ...
  plug MyApp.CacheAPIPlug
  ...
end

Cache the response after sending it:

def some_api_action(conn, _params) do
  # Process the request and generate the response
  response = %{data: "Some API response data"}

  # Convert the response to JSON
  json_response = Jason.encode!(response)

  # Cache the response
  cache_key = "api_cache:#{conn.request_path}?#{conn.query_string}"
  cache_value = {200, [{"content-type", "application/json"}], json_response}
  Cachex.put(:my_cache, cache_key, cache_value)

  # Send the response
  json(conn, response)
end

Checking for Stale Cache Data

One of the main challenges with caching is ensuring that the data in the cache is up-to-date. If data becomes stale, it can lead to incorrect results and a poor user experience. Therefore, it's crucial to check for stale data and refresh the cache when necessary.

Cachex provides built-in mechanisms to handle stale data, primarily through its Time-to-Live (TTL) and fallback features.

TTL

TTL allows you to specify how long an item should remain in the cache. Once the TTL expires, an item is automatically removed from the cache. This ensures that you don't serve stale data older than a specified age.

# Store data with a TTL of 3600 seconds (1 hour)
Cachex.put(:my_cache, "key", "value", ttl: 3600)

When you attempt to retrieve this key after its TTL has expired, it will return as if the key does not exist in the cache.

Fallbacks

Cachex's fallback mechanism is a powerful feature that allows you to execute a function when a cache miss occurs. This can be especially useful for handling stale data. If data is not in the cache (either because it was never cached or because it was evicted due to TTL expiration), the fallback function can fetch fresh data.

fallback_fn = fn key ->
 # Fetch fresh data for the given key
 {:commit, fetch_fresh_data(key)}
end

# Attempt to get data from cache, use fallback if cache miss occurs
Cachex.fetch(:my_cache, "key", fallback: fallback_fn)

The :commit tuple ensures that fetched data is stored back into the cache.

By combining TTL and fallbacks, Cachex provides a robust mechanism to ensure that stale data is not served and that fresh data can be fetched and cached automatically when needed.

One thing to note is the difference between Cachex.get and Cache.fetch. The Cachex.get function returns the value associated with the given key, or {:ok, nil} if the key is not found in the cache. The Cachex.fetch function returns the value associated with the given key, or executes the fallback function if the key is not found in the cache. While both functions can be used to retrieve data from the cache, they have different behaviors when the key is not found. Cachex.fetch offers more advanced mechanisms to handle cache misses.

It's important to understand that the TTL and fallback features are not mutually exclusive, and can be used together to provide a more robust caching solution and a better user experience.

Wrapping Up

In this post, we've seen that caching is a powerful technique to significantly improve your Elixir application's performance. However, we've also explored how caching is not without its drawbacks.

Cachex, as representative of the vibrant Elixir ecosystem, showcases how community-driven tools can address complex problems with elegance and efficiency. But remember, Cachex is just the tip of the iceberg. The Elixir community is teeming with innovative libraries and frameworks, each solving unique challenges and pushing the boundaries of what's possible.

As you continue your journey with Elixir, I encourage you to explore the many tools and libraries available and discover how they can help you build better applications.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Creating Custom Exceptions in Elixir

Roy Tomeij — 2024-02-20T00:00:00+00:00

Exceptions and exception handling are widely accepted concepts in most modern programming languages. Even though they're not as prevalent in Elixir as in object-oriented languages, it's still important to learn about them.

In this article, we will closely examine exceptions in Elixir, learning how to define and use them.

Let's get started!

Elixir and Exceptions

There is no single golden rule that covers when to use exceptions, especially custom ones. Throughout this article, I will keep to a definition of exceptions from StackOverflow:

An exception is thrown when a fundamental assumption of the current code block is found to be false.

So, we can expect an exception when something truly exceptional and hard to predict happens. Network failures, databases going down, or running out of memory — these are all good examples of when an exception should be thrown. However, if the form input you send to your server does not pass validation, there are better expressions to use, such as error tuples.

Check out our An Introduction to Exceptions in Elixir post for an overview of exceptions.

You might wonder how Erlang's famous "let it crash" philosophy works with exceptions. I would say it works pretty well. Exceptions are, in fact, crashes, as long as you don't catch them.

The Anatomy of Elixir's Exceptions

The most common exception you may have seen is probably NoMatchError. And if you've used Ecto with PostgreSQL, you also must have seen Postgrex.Error at least a few times. Among other popular exceptions, we have CaseClauseError, UndefinedFunctionError, or ArithmeticError.

Let's now take a look at what exceptions are under the hood. The easiest way to do that is to cause an exception, rescue it, and then inspect it. We will use the following code to dissect NoMatchError:

defmodule Test do
  def test(x) do
    :ok = x
  end
end

try do
  Test.test(:not_ok)
rescue
  ex -> IO.inspect(ex)
end

The output will be:

%MatchError{term: :not_ok}

As we can see, this is just a struct with some additional data. Using similar code, we can check CaseClauseError.

%CaseClauseError{term: :not_ok}

We can do more using functions provided by the Exception module:

> Exception.exception?(ex) # note that this is deprecated in favour of Kernel.is_exception
true
> Exception.message(ex)
"no case clause matching: :not_ok"
> Exception.format(:error, ex)
"** (CaseClauseError) no case clause matching: :not_ok"

And can peek even deeper by using functions from the Map module:

> Map.keys(ex)
[:__exception__, :__struct__, :term]
> ex.__struct__
CaseClauseError
> ex.__exception__
true
> ex.term
:not_ok

Armed with that knowledge, we create a "fake" exception using just a map:

> Exception.format(:error, %{__struct__: CaseClauseError, __exception__: true, term: :not_ok})
"** (CaseClauseError) no case clause matching: :not_ok"

However, this is not how we will define our custom exception. And before we dive into custom exceptions, let's try to answer one important question: when should we use them?

When Should You Use Custom Exceptions?

The most common use case for custom exceptions is when you are creating your own library. Take Postgrex, for example: you have Postgrex.Error. In Tesla (an HTTP client), you have Tesla.Error. They are useful because they immediately indicate where an error happens and how to determine its cause.

Most of us, however, do not write libraries often. It's more likely that you work on a specific application that powers your company's business (or its customers). Even in your application's code, it can be useful to define some custom exceptions.

For example, your application might send webhook notifications to a URL defined in an environment variable. Consider this very simplified code:

defmodule WebhookSender do
  def send(payload) do
      url = System.get_env("WEBHOOK_ENDPOINT")
      HttpClient.post(url, payload)
  end
end

You can reasonably expect that a WEBHOOK_ENDPOINT environment variable is set on a machine where your application is deployed. If it's not, that's a misconfiguration. To paraphrase the earlier definition of exceptions from StackOverflow, it's a "fundamental assumption of the current code being false".

Of course, running that code when System.get_env call evaluates to nil will result in an exception from HttpClient. However, instead, you can be more defensive and perform a direct check in the WebhookSender module.

Imagine you swap HttpClient for BetterHttpClient in the future, and all exceptions change. Now, throughout your application, you must fix all the places where you use a reported exception (for example, when providing an informative error message to the client). And this is because you changed a dependency, an implementation detail.

How to Define a Custom Exception in Elixir

As we have seen, exceptions in Elixir are just "special" structs. They are special because they have an __exception__ field, which holds a value of true. While we could just use a Map of regular defstruct, this exception would not work nicely with all the tooling around exceptions.

To define a proper exception, we should use the defexception macro. Let's do this for the webhook example we looked at earlier:

defmodule WebhookSender.ConfigurationError do
  defexception [:message]
end

It is as simple as that. You can then improve the code of the WebhookSender:

defmodule WebhookSender do
  def send(payload) do
      case System.get_env("WEBHOOK_ENDPOINT") do
        nil -> raise ConfigurationError, message: "WEBHOOK_ENDPOINT env var not defined"
        url -> HttpClient.post(url, payload)
      end
  end
end

If you run this code (of course, assuming the variable is not set), it will show an error just like with a regular exception:

** (WebhookSender.ConfigurationError) WEBHOOK_ENDPOINT env var not defined
    exceptions_test.exs:24: (file)
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2

This exception clearly shows where the error originated from: a WebhookSender module. Imagine that, instead, you see something like HttpClient.CannotPerformRequest here. Chances are you are using HttpClient in multiple places in the application. First, you must traverse the stack trace and find out which HttpClient invocation is the culprit. Then, you still have to figure out the actual reason for the error.

Note that :message is just an example of a field you can define on the exception. Although it's a nice default, it is not strictly needed.

defmodule SpaceshipConstruction.IncompatibleModules do
  defexception [:module_a, :module_b]
end

raise SpaceshipConstruction.IncompatibleModules,
  module_a: LithiumLoadingBay,
  module_b: OxygenTreatmentPlant

When this code runs, however, it will crash on attempting to format the exception message:

** (SpaceshipConstruction.IncompatibleModules) got UndefinedFunctionError with message "function SpaceshipConstruction.IncompatibleModules.message/1 is undefined or private" while retrieving Exception.message/1 for %SpaceshipConstruction.IncompatibleModules{module_a: LithiumLoadingBay, module_b: OxygenTreatmentPlant}. Stacktrace:
    SpaceshipConstruction.IncompatibleModules.message(%SpaceshipConstruction.IncompatibleModules{module_a: LithiumLoadingBay, module_b: OxygenTreatmentPlant})
    (elixir 1.14.0) lib/exception.ex:66: Exception.message/1
    (elixir 1.14.0) lib/exception.ex:117: Exception.format_banner/3
    (elixir 1.14.0) lib/kernel/cli.ex:102: Kernel.CLI.format_error/3
    (elixir 1.14.0) lib/kernel/cli.ex:183: Kernel.CLI.print_error/3
    (elixir 1.14.0) lib/kernel/cli.ex:145: anonymous fn/3 in Kernel.CLI.exec_fun/2

    space_exceptions.exs:5: (file)
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2

There are two ways to fix this without having to manually pass an error message every time:

Add a message/1 function to an exception module.

defmodule SpaceshipConstruction.IncompatibleModules do
  defexception [:module_a, :module_b]
  def message(_), do: "Given modules are not compatible"
end

Here, an argument to the function is an exception itself, so you can construct a more precise error message from it. For example:

def message(exception) do
  "Module #{exception.module_a} and #{exception.module_b} are not compatible"
end

Provide a default message.

defmodule SpaceshipConstruction.IncompatibleModules do
  defexception [:module_a, :module_b, message: "Given modules are not compatible"]
end

Repackaging Exceptions

One interesting use case for custom exceptions is when you want to "repackage" an existing exception to fit a specific condition. This can make the exception stand out more in your error tracker.

For example, we did that with database deadlocks at the company I work for. Deadlocks are one of the hardest database-related errors to track, but they are just reported as Postgrex.Error. We wanted clearer visibility over when these errors happen (compared to other Postgrex.Errors) and on which GraphQL mutations.

So, I added an Absinthe middleware checking for the exception. It looks like this:

defmodule MyAppWeb.Middlewares.ExceptionHandler do
  alias Absinthe.Resolution
  @behaviour Absinthe.Middleware

  def call(resolution, resolver) do
    Resolution.call(resolution, resolver)
  rescue
    exception ->
      error = Exception.format(:error, exception, __STACKTRACE__)

      if String.match?(error, ~r/ERROR 40P01/) do
        report_deadlock(exception, __STACKTRACE__)
      else
          @error_reporter.report_exception(exception, stacktrace: __STACKTRACE__)
      end

    resolution
  end

  defp report_deadlock(ex, stacktrace) do
      original_message = Exception.message(ex)
      mutation = get_mutation_name_from_process_metadata()
      try do
      reraise DeadlockDetected,
        [message: "Deadlock in mutation #{mutation}\n\n#{original_message}"],
        stacktrace
    rescue
      exception ->
        @error_reporter.report_exception(exception,
          stacktrace: __STACKTRACE__
        )
    end
  end
end

This might seem like a lot of code. Let's break it down a little. When an exception is raised, we check if its message contains ERROR 40P01 (code for a deadlock).

Then, we raise a custom DeadlockDetected exception and immediately rescue it to send it to an error reporter, such as AppSignal.

Now, instead of generic Postgrex.Errors, often mixed with other database exceptions, we have a separate class of exceptions just dedicated to deadlocks. And custom exception messages allow us to quickly identify the mutation with deadlock-unsafe code.

Repackaging Exits as Exceptions

Another case for custom exceptions is when you want to transform an exit into an exception. This might be because your error reporting software does not support exits, or you may just want a more specific message than the default.

The most common case for exits is timeouts:

defmodule ImportantTask do
  def run do
    task = Task.async(fn -> :timer.sleep(200) end)
    Task.await(task, 100)
  end
end

ImportantTask.run()

In the above code, we spawn an async task that takes 200 milliseconds to complete, but we allow it to run for 100 ms. Here's the result:

** (exit) exited in: Task.await(%Task{mfa: {:erlang, :apply, 2}, owner: #PID<0.96.0>, pid: #PID<0.103.0>, ref: #Reference<0.131053822.3597205508.67962>}, 100)
    ** (EXIT) time out
    (elixir 1.14.0) lib/task.ex:830: Task.await/2
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2

It's pretty generic, isn't it? Let's make it a bit nicer.

defmodule ImportantTask do
  defmodule Timeout do
    defexception [:message]
  end

  def run do
    task = Task.async(fn -> :timer.sleep(200) end)
    Task.await(task, 100)
  catch
  :exit, {:timeout, _} = reason ->
      error = Exception.format_exit(reason)
      raise Timeout, message: error
  end
end

ImportantTask.run()

Here, we define a custom Timeout exception, then catch an exit and raise an exception instead. The result is:

** (ImportantTask.Timeout) exited in: Task.await(%Task{mfa: {:erlang, :apply, 2}, owner: #PID<0.96.0>, pid: #PID<0.106.0>, ref: #Reference<0.342776.2255814665.42176>}, 100)
    ** (EXIT) time out
    exits_to_exceptions.exs:12: ImportantTask.run/0
    (elixir 1.14.0) lib/code.ex:1245: Code.require_file/2

While you may consider that this error message is still a bit cryptic, it adds two main quality-of-life improvements:

An ImportantTask.Timeout exception, which makes it easy to assign the error to a particular piece of functionality in your code.
A line from our code in the stack trace (exits_to_exceptions.exs:12: ImportantTask.run/0). Note that the default exit message does not include this, so it's much harder to find the offending place in the code.

Wrapping Up

In this post, we learned how to define custom exceptions in Elixir. They are very useful when building a library, but they also have their place in your application code.

By repackaging generic exceptions or trapping and re-raising exits, you can make your code much easier to debug if something goes wrong. Your future self (and your colleagues) will be grateful!

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Build a Memory-efficient Elixir App with Streams

Roy Tomeij — 2024-02-06T00:00:00+00:00

We have all encountered collections of data at some point when working on Elixir applications. These collections are very handy for storing, retrieving, and manipulating data using different data structures, making them very efficient in managing clean code.

In this article, we'll go through the following:

What are collections in Elixir?
The problems likely to be faced when working with collections
The greedy approach when working with large datasets
The enumerable in Elixir, used to work around collections

Finally, we'll explore how to build a memory-efficient Elixir application using the lazy processing approach with streams.

What are Collections in Elixir?

You can view a collection as a container that groups multiple things/objects/elements into a single unit. This collection allows you to store, retrieve, and manipulate data.

A couple of examples of collections encountered in the real world include:

A mail folder that contains received and sent letters.
A phone directory mapping phone numbers to corresponding names.

In Elixir, List, Maps, Ranges, files, and tuples are the most common types of collections. Some of these collections can be iterated through. Each element is passed through once, in order, so we can term them enumerables. Things that can be iterated in Elixir implement the Enumerable protocol. List, Maps, and Ranges are the most common datatypes used as enumerables.

Elixir has two modules with iteration functions that use the Enumerable protocol: Enum and Stream.

The Challenges of Collections

Working with collections can be very efficient, especially when your collections aren't that large. However, manipulating these collections can be very difficult when dealing with large datasets.

For example, let's say we want to retrieve organization data from a CSV file and display it in our application. Ideally, we have to load all the organizations from the CSV file into an application's memory and transform the retrieved data into the data we want.

Our application can be safe if the memory contains a predictable number of organizations that can be easily transformed or manipulated. However, if we have extensive data on organizations retrieved from the CSV, managing this data can result in slow application performance, because more memory is needed to consume it.

So, how can we elegantly manage our large CSV file? (keeping in mind some of the problems we might face while processing it).

We can process our CSV file using one of two approaches:

The greedy approach that consumes more memory.
The memory-efficient lazy processing approach.

Earlier, we mentioned that Elixir has two modules, Enum and Stream, with iteration functions. We'll use these modules to process our CSV file.

The Greedy Approach For Our Elixir Application

In the greedy approach, we are going to use the Enum module because:

When given a collection, it will consume all the content of that collection. That means the whole collection should be loaded into memory before being processed.
The returned result is another collection that will be loaded into memory, which might be inefficient if data is not needed at that time.

Let's process our CSV file greedily.

We shall use an organization's CSV file with 2,000,000 records downloaded from here for demonstration purposes. The size of the CSV file (after unzipping) is around 283.4MB.

Our focus will be to get the records of all organizations and transform them into data we can easily display in the browser. The result should look like this:

[
%{
index: "1",
organization_id: "1234",
name: "PLC",
website: "https://park.com/",
country: "Kenya",
description: "Focused for the best",
founded: "2016",
industry: "sports",
num_of_employees: 300

},..
]

Before we begin, let's open the interactive shell to manipulate the CSV file. After that, start the Erlang Observer application by running:

iex > :observer.start

We shall use the Observer application to inspect memory allocators when processing the CSV file.

Step 1: Loading the CSV File Into Memory

Use File.read!("organization.csv") to load the whole CSV file into memory. The observer shows an extra 283.84 MB allocated to memory.

iex > org_csv = File.read!("organization.csv")


    "Index,Organization Id,Name,Website,Country,Description,Founded,Industry,Number of employees\r\n1,391dAA77fea9EC1,Daniel-Mcmahon,https://stuart-rios.biz/,Cambodia,Focused eco-centric help-desk,2013,Sports,1878\r\n2,9FcCA4A23e6BcfA, Mcdowell,http://jacobs.biz/,Guyana,Front-line real-time portal,2018,Legal Services,9743\r\n3,DB23330238B7B3D,\"Roberts, Carson and Trujillo\",http://www.park.com/,Jordan,Innovative hybrid data-warehouse,1992,Hospitality,7537\r\n4,bbf18835CFbEee7,\"Poole, Jefferson and Merritt\",http://hayden.com/,Cocos (Keeling) Islands,Extended regional Graphic Interface,1991,Food Production,9974\r\n5,74ECD725ceaDfd9,\"Ritter, Patel and Cisneros\",https://www.mason-blackwell.info/,Ecuador,Re-contextualized actuating website,2019,Computer Networking,5050\r\n6,ea42e2FECDAdF0c,Stafford Ltd,http://www.fuller.biz/,Qatar,Multi-channeled optimizing customer loyalty,1979,Aviation / Aerospace,7506\r\n7,b8EE5AE2Df8BA46,Roach Ltd,https://www.oconnell.com/,Guatemala,Switchable explicit complexity,2020,Hospitality,109\r\n8,21829Cf5a968024,Gill PLC,https://www.berry.com/,Lesotho,Future-proofed systemic pricing structure,1983,Printing,1822\r\n9,f9C605c034f47cD,Summers-Jordan,https://le.net/,Luxembourg,Cross-platform bottom-line system engine,1976,Primary / Secondary Education,1603\r\n10,0fBa55ecDA6cb4B,\"Richard, Lane and Weaver\",https://bauer-hardy.com/,United States Minor Outlying Islands,Optimized system-worthy complexity,2006,Building Materials,3505\r\n11,d085befF19Be6fB,\"Harrington, Sutton and Wilkins\",http://henson.com/,Guinea-Bissau,Diverse scalable instruction set,1985,Airlines / Aviation,1926\r\n12,13B77fAF602E949,Evans LLC,http://harrington-powers.com/,Cook Islands,Re-contextualized stable flexibility,2017,Design,3338\r\n13,90234C8d0D7eB75,\"Patterson, Deleon and Donovan\",https://thornton.net/,Holy See (Vatican City State),Versatile encompassing migration,1970,Automotive,9312\r\n14,155A7db5D8Cce47,\"Carlson, Snyder and Holland\",http://foley.com/,Korea,Devolved optimal secured line,1979,International Trade / Development,2649\r\n15,77B7F7fb1Ac2A44,Duffy-Stark,https://www.morales.com/,Monaco,Re-contextualized 24/7 Graphical User Interface,1990,Aviation / Aerospace,1714\r\n16,1f6bABBA98cd33E,\"Frederick, Fry and Poole\",http://www.madden.org/,Vanuatu,Synchronized empowering structure,2008,Fishery,1862\r\n17,05dbf87Ee09b6BC,\"Walton, Proctor and Peters\",http://www.casey-bell.com/,Grenada,Diverse local collaboration,1979,Alternative Medicine,4678\r\n18,20ef50A2fdB3993,Oneal and Sons,http://munoz.org/,Fiji,Operative uniform definition,1987,Religious Institutions,8332\r\n19,a9C703D1A2B074C,Maynard-Bush,http://leblanc.com/,Senegal,Customizable zero tolerance functionalities,2011,Apparel / Fashion,1668\r\n20,1ea33Ac2face255,\"Rosario, Martin and White\",https://www.mueller-rhodes.net/,Isle of Man,Synchronized 6thgeneration flexibility,1994,Music,5134\r\n21,CDCAe4a6DA0eC6d,Brady Ltd,http://montgomery.biz/,Guyana,Self-enabling modular archive,1989,Professional Training,5153\r\n22,f41b9Ce917bDD28,Odom Group,http://www.riley-dalton.net/,Saint Kitts and Nevis,Grass-roots bandwidth-monitored projection,2003,Computer Hardware,4793\r\n23,B1181c8C3d43216,Stanton-Arroyo,http://baird.org/,Montserrat,Integrated empowering core,1976,Mental Health Care,6919\r\n24,b3E9C4E4cbC4a8e,Stafford-Hickman,https://love.net/,United States Virgin Islands,Virtual background application,1996,Hospital / Health Care,3855\r\n25,f92AacaEEcB6eC6,Ryan and Sons,http://koch-raymond.org/,Chad,Public-key maximized definition,2017,Airlines / Aviation,1564\r\n26,8eb91E6eeDBC2A5,Wolf Group,https://aguilar-solomon.net/,Cayman Islands,Public-key value-added alliance,1984,Package / Freight Delivery,4509\r\n27,fDaeD26dB6fd79a,\"Horne, Bailey and Oconnor\",https://cruz.com/,Western Sahara,Customer-focused needs-based service-desk,1973,Wholesale,2670\r\n28,19Bbb8dB90dFaFF,Cobb Inc,http://miles.com/,Netherlands,Assimilated local ability,2008,Political Organization,2494\r\n29,00AF7EacF3fEb91,Parrish-Peterson,https://strong.org/,France,Multi-lateral encompassing structure,2004,Political Organization,9059\r\n30,8Ec27fFD6b945A9,\"Barr" <> ...

Step 2: Create a List for Each Row In the CSV

The data in Step 1 is loaded text, which means we are still far from our end goal. In this step, let's take the loaded text data and split it into lines. This will result in a list of string elements, with each element representing a row of the CSV.

iex > [_header | rows] = org_csv |> String.split("\n")

     ["Index,Organization Id,Name,Website,Country,Description,Founded,Industry,Number of employees\r",
 "1,391dAA77fea9EC1,Daniel-Mcmahon,https://stuart-rios.biz/,Cambodia,Focused eco-centric help-desk,2013,Sports,1878\r",
 "2,9FcCA4A23e6BcfA,Mcdowell, http://jacobs.biz/,Guyana,Front-line real-time portal,2018,Legal Services,9743\r",
...]

The observer shows an increase in allocated memory:

Step 3: Create a Column from Each Row

In the second step, we pattern-matched our result to [_header | rows]. We are not interested in the header of the CSV file, but in the body.

In this step, we will split each row into a list, with the elements in each list representing a column of the CSV file.

iex > org_records = rows |> Enum.map(&String.split(&1, ","))

  [
    ["1", "391dAA77fea9EC1", "Daniel-Mcmahon", "https://stuart-rios.biz/",
     "Cambodia", "Focused eco-centric help-desk", "2013", "Sports", "1878\r"],
    ["2", "9FcCA4A23e6BcfA", "Mcdowell",
     "http://jacobs.biz/", "Guyana", "Front-line real-time portal", "2018",
     "Legal Services", "9743\r"],
...]

This step alone takes almost 60 seconds to process. The observer also shows a spike increase in memory allocation:

Step 4: Create a Keyword List of Each Row

Here, each column in each row will be zipped to its corresponding header name.

iex > org_records = org_records |> Enum.map(fn record ->  Enum.zip([:org_id, :name, :website, :country, :description, :founded, :industry, :number_of_employees], record) end)
    [
      [
       index: "1",
       org_id: "391dAA77fea9EC1",
       name: "Daniel-Mcmahon",
       website: "https://stuart-rios.biz/",
       country: "Cambodia",
       description: "Focused eco-centric help-desk",
       founded: "2013",
       industry: "Sports",
       number_of_employees: "1878"
      ],
      [
        index: "2",
        org_id: "9FcCA4A23e6BcfA",
        name: "Mcdowell",
        website: "http://jacobs.biz/",
        country: "Guyana",
        description: "Front-line real-time portal",
        founded: "2018",
        industry: "Legal Services",
        number_of_employees: "9743"
      ],
...]

In this case, we use Enum.map/2 and Enum.zip/2 to process the organization record results from step 3 (a list of lists, where each list element represents a row of records). Remember that each call to the Enum module takes a collection and returns a collection. Both Enum.map/2 and Enum.zip/2 will greedily process our data, which explains the spike increase in memory allocation as seen from the observer.

Enum.map/2 takes in a list of lists, where each list element is a row of organization records. It iterates through each corresponding row and invokes Enum.zip/2 on each row to transform it into a keyword list.

The Enum.zip/2 function takes in two enumerables. The first enumerable is a list of header names, and the second is a list of each row. The zip function zips the corresponding elements from the list of headers and the list of each row into a list of tuples. The tuple element in the list has an atom as the first element (which is the header name and value of the corresponding header name).

Elixir represents this kind of output as a keyword list. In your interactive shell, you will notice each row is transformed into a keyword list, but not a list of tuples.

Enum.map/2 returns a new list of keyword lists.

Step 5: Convert to Map

Remember, our end goal was to work with data that's easy to manipulate when we want to display a list of organizations. In this final step, we will convert the keyword list above into maps.

iex > org_records |> Enum.map(fn data -> Map.new(data) end)

    [
      %{
        index: "1",
        org_id: "391dAA77fea9EC1",
        name: "Daniel-Mcmahon",
        website: "https://stuart-rios.biz/",
        country: "Cambodia",
        description: "Focused eco-centric help-desk",
        founded: "2013",
        industry: "Sports",
        number_of_employees: "1878"
      },
      %{
        index: "2",
        org_id: "9FcCA4A23e6BcfA",
        name: "Mcdowell",
        website: "http://jacobs.biz/",
        country: "Guyana",
        description: "Front-line real-time portal",
        founded: "2018",
        industry: "Legal Services",
        number_of_employees: "9743"
      },
...]

Compiling everything together:

iex > [_| rows ] =
  "organization.csv"
 |>  File.read!()
 |>  String.split("\n")
 |> Enum.map(&String.split(&1, ","))


iex >
   rows
|> Enum.map(fn record ->  Enum.zip([:org_id, :name, :website, :country, :description, :founded, :industry, :number_of_employees], record) end)
|> Enum.map(fn record -> Map.new(record) end)

Observation with `Enum`

In each Enum step, we generated an output before the final result. The intermediate result in each step is stored in memory as a full collection. That's because, with Enum implementation, the whole collection must be available for the next step in the pipeline to start processing.
With Enum, each step has to wait for its intermediate result to start processing, so it takes more time to process data in each step.
There is a spike increase in memory allocation in each step, meaning an intermediate output must be kept in memory.

The greedy approach, or working with Enum, works well with a predictable number of elements. We couldn't have easily noticed these problems in a small CSV file.

However, this approach is very inefficient when we have to keep an intermediate result in memory in each step. This is a massive waste of memory that can lead to slow application performance.

Solution: Build a Memory-efficient Elixir App with Streams

First, what are our goals?

Memory allocation that's only for the final result.
To process records only when we need them.

Using streams, we can do these things. Streams are known to be lazy enumerables.

This is because streams:

Lazily load data into memory: That is, one element at a time instead of loading everything at once.
Only load data when needed.
Instead of applying a transformation to the given enumerable, return a value with the intended specifications.
Process one piece of data at a time as it arrives instead of waiting for the whole collection to be available.
Are a composable enumerator that reads and pipes one single line at a time without needing to wait for all the passed data to be processed at every single step.

The Lazy Processing Approach with Streams in Elixir

We can leverage lazy processing to process our organization's CSV file.

The good thing is that more Elixir modules now support streams. Instead of opening the CSV file and loading the data on step 1, let's use File.stream!/3 to create a stream without necessarily opening the file.

iex > File.stream!("/org.csv")
    %File.Stream{
      line_or_bytes: :line,
      modes: [:raw, :read_ahead, :binary],
      path: "/org.csv",
      raw: true
    }

The File.stream!/3 function returns %File.Stream{} with intended specifications.

The observer doesn't reflect our CSV file's extra memory size, meaning the file still needs to be opened and loaded into memory.

Streams are composable enumerables. We can pipe one stream to another. We will replace the Enum functions from steps 2-4 of the greedy approach with stream functions. We'll avoid processing data and loading it into memory until we decide to run the stream (compute/process the data as intended) by passing it to a function in the Enum module.

Let's compose our first part of the pipeline with streams:

iex > header = [:index, :org_id, :name, :website, :country, :description, :founded, :industry, :number_of_employees]
iex >
"/org.csv"
|> File.Stream!()
|> Stream.drop(1)
|> Stream.map(&String.split(&1, ["\n", ","], trim: true))
|> Stream.map(&Stream.zip(header, &1))
|> Stream.map(&(Map.new(&1)))

  #Stream<[
  enum: %File.Stream{
    path:  "/org.csv",
    modes: [:raw, :read_ahead, :binary],
    line_or_bytes: :line,
    raw: true,
    node: :nonode@nohost
  },
  funs: [#Function<34.53678557/1 in Stream.drop/2>,
   #Function<48.53678557/1 in Stream.map/2>,
   #Function<48.53678557/1 in Stream.map/2>,
   #Function<48.53678557/1 in Stream.map/2>]
]>

The composed pipeline of streams above returns a stream that only stores the intended computation instead of applying the transformation.

Here is what is happening within our stream processing pipeline:

Instead of opening the file, we use File.Stream/3 to create a stream.
Stream.drop/2 lazily drops the first element, the header.
Stream.map/2:
- splits data into columns
- lazily zips each row into its corresponding header name with Stream.zip/2
- transforms each piece of data into a map

To process our desired result, we have to pass the stream value to a function in the Enum module. First, let's take only the first 3 elements from our stream and observe what happens.

iex > csv
     |> File.stream!()
     |> Stream.drop(1)
     |> Stream.map(&String.split(&1, ["\n", ","], trim: true))
     |> Stream.map(&Stream.zip(header, &1))
     |> Stream.map(&(Map.new(&1)|> IO.inspect(label: "=============")))
     |> Enum.take(3)

Enum.take/2 receives the transformed map data from the stream and only gets the first 3 elements. Once it gets the 3 elements from the stream, there is no more processing.

Running this pipeline, you will notice that:

The result is returned instantaneously.
IO.inspect only prints the first 3 elements. This is proof that the elements are being processed one at a time by subsequent calls to Enum functions.

Lazy Approach Using Streams Vs. Greedy Approach

Let's use the greedy approach to take the first 3 elements from our code:

iex >
[_| rows ] =
  csv
 |>  File.read!()
 |>  String.split("\n")
 |> Enum.map(&String.split(&1, ","))


iex >
   rows
|> Enum.map(fn record ->  Enum.zip([:org_id, :name, :website, :country, :description, :founded, :industry, :number_of_employees], record) end)
|> Enum.map(fn record -> Map.new(record) |> IO.inspect(label: "+++++++++++++++++") end)
|> Enum.take(3)

IO.inspect prints the whole 2,000,000 records. This shows that with Enum, we have to wait for the whole collection to be available upfront before starting the next processing batch. Using this approach, an intermediate list of 2,000,000 records has to be kept in memory at each step (even if, in the end, we only get the first 3 records using Enum.take/2).

Unlike the greedy approach, using the lazy approach, there is no need to process the whole collection for the next process to take place. By passing Enum.take/2 to fetch the first 3 elements, we can see there isn't even a peak in memory.

With streams, even if we process 2,000,000 records, we can decide the amount of records to load in memory and ensure that the amount of utilized memory stores the final result. This is not the same case with Enum, since in each step there is a certain amount of records kept in memory.

We can also compare the greedy and lazy approaches by monitoring memory allocators using the Erlang observer.

Remember, in the greedy approach, we processed all records. We will do the same with our lazy approach by passing the stream pipeline to Enum.to_list/1.

iex > csv
     |> File.stream!()
     |> Stream.drop(1)
     |> Stream.map(&String.split(&1, ["\n", ","], trim: true))
     |> Stream.map(&Stream.zip(header, &1))
     |> Stream.map(&(Map.new(&1)|> IO.inspect(label: "=============")))
     |> Enum.to_list()

Here's the memory allocated to the lazy approach:

And to the greedy approach:

Wrapping Up

In this post, we first took a look at collections in Elixir and some of their associated challenges. We then explored two methods of processing large datasets: the greedy approach and the lazy approach using streams.

We've seen how working with streams is one of the best approaches to help create a memory-efficient Elixir application.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Build A Simple Tracing System in Elixir

Roy Tomeij — 2024-01-23T00:00:00+00:00

In this post, we'll cover how Elixir applications can be traced using OpenTelemetry and how macros can make this process super easy and streamlined.

First, we'll talk about tracing and OpenTelemetry in Elixir. Then we'll improve our custom tracing layer step-by-step until we get an easy and seamless tool to trace our application.

Let's get started!

On Tracing in Elixir

Note: All the research done to write this article resulted in the creation of the abstracing library. It's far from complete, but it encapsulates all the ideas written here.

Think of instances when your app crashes in production. A bunch of artifacts are generated: stacktraces, logs, reports, etc.

When using proper tracing, developers can link all these artifacts in a sequence of events — from a starting point down to the response, and operations with side effects.

Here is a simple example of a POST /users trace request:

From left to right, we can see that the request was received, decoded, validated, saved, and then, finally, a response was encoded and sent. Each little block in this trace is called a span. Spans are the building blocks of tracing, as they represent events inside an application.

Spans require this basic information: the start, the end, and the status (success or error). We can enrich each span with more data, and they can even have a direct correlation to these entities if we add the appropriate metadata to logs and errors.

Meet OpenTelemetry for Elixir

The OpenTelemetry project homepage states that it is a:

Collection of APIs, SDKs, and tools

That can be used to:

Instrument, generate, collect, and export telemetry data (metrics, logs, and traces).

For Elixir, the OpenTelemetry library has everything we need to perform distributed tracing.

Here is a very simple example of how we can use it:

# example.ex
defmodule Example do
  require OpenTelemetry.Tracer

  def some_fun() do
    OpenTelemetry.Tracer.with_span "example.some_fun" do
      # ...
      OpenTelemetry.Tracer.set_attribute("key", "value")
      # ...
  end
end

What's Happening Here?

As you can see, it's pretty straightforward. To start using all macros inside OpenTelemetry.Tracer, we first require it at the top of our module.

When we want to start a span, we just need to call OpenTelemetry.Tracer.with_span/2 and write our code.

Under the hood, :otel_tracer.with_span/4 is used to actually start the span — even though the OpenTelemetry API does provide Elixir modules to interact with, all the heavy lifting is actually written in Erlang.

One really cool thing about spans is that we can add metadata to reported data. This gives us more contextual information to investigate issues. We can do this by calling OpenTelemetry.Tracer.set_attribute/2. It only accepts a small set of types (atoms, booleans, binaries, and tuples), so we need to be mindful when using it.

Now for a brief overview of how I ended up building an abstraction layer for OpenTelemetry.

Why I Built an Abstraction Layer for OpenTelemetry

When I first started using OpenTelemetry, I noticed that I was constantly creating small private functions to translate data and help me with the setup. As I did more and more of that, I eventually extracted all this boilerplate to its own feature. I created an abstraction layer for OpenTelemetry.

A few pain points during my use of OpenTelemetry that were solved by this abstraction layer:

If your code throws an unexpected exception, the span will not be collected.
Adding complex data requires transforming it first.
Long namespaces (not really a problem, but I just don't like them 😁).

Luckily, the OpenTelemetry library also has a low-level API that can be used to customize our tracing tooling, and that's exactly what we'll be doing now!

Breaking Down the Pain Points of OpenTelemetry

Now that we know some of the direct pain points of using OpenTelemetry, let's break them down into separate categories and solve each one. We can group the features that we inject boilerplate code in to:

Setup: how we prepare a module to be traced
Start/stop spans: the steps required to actually create spans
Modifying spans: adding more attributes to spans
Exception handling: collecting errors and changing the span status

In the end, we want to cover all these features with the least amount of boilerplate code as possible.

A Setup for a Setup in Elixir

The very first thing we need to do is prepare our module to use the tracing macros and libraries. We'll use Elixir's special macro __using__/1 to automate some of this stuff for us:

# lib/tracing.ex
# ...
defmacro __using__(_opts) do
  quote do
    require OpenTelemetry.Tracer

    require unquote(__MODULE__)
    import unquote(__MODULE__), only: [span: 1, span: 2, span: 3]
  end
end
# ...

Now, whenever we need to trace our code, we just need to call use Tracing at the top of our module.

defmodule MyModule do
  use Tracing
  # ...
end

So far, so good — but nothing too exciting. However, by using this simple setup macro we don't have to modify the modules using it if we ever make changes to the setup process, as all changes will be automatically replicated. That's a good start!

In the next section, we'll start to remove some more meaningful boilerplate.

Translating Elixir Application Data to Span Attributes

OpenTelemetry allows applications to include attributes in spans. An attribute consists of a key and a value. This helps us to include useful information that can later be used to either create monitoring triggers or investigate a crash.

But here is a catch: OpenTelemetry only accepts numbers, strings, atoms, booleans, and lists (if its elements are from any of the supported basic types). Applications work with a richer set of data types: not only numbers and strings but also complex lists, maps, structs, and tuples.

Of course, we can use inspect/1 on the variable values and have everything in there. However, this makes searching for spans a much harder task, as we would need to use complex regexes to search for them.

Convert Complex Types to Basic Types

It's possible, however, to convert the complex types to more basic (and supported) types. Let's define a few rules:

Lists will use their indexes to name the values
Tuples will be converted to lists
Maps will use their keys to name values
Structs will be converted to maps

So, a simple list like [1, 2, 3] would be transformed into a list of pairs:

my_list = [1, 2, 3]
Tracing.set_attribute("numbers", my_list)
# numbers.0 = 1
# numbers.1 = 2
# numbers.2 = 3

For maps, we can use their keys to generate the pairs:

my_map = %{first_name: "John", last_name: "Wick"}
Tracing.set_attribute("user", my_map)
# user.first_name = "John"
# user.last_name = "Wick"

Using Elixir's `defguard`

Since we're handling maps, lists, and tuples as a set of values, we can use Elixir's defguard to create a custom function guard:

# lib/tracing.ex
defmodule Tracing do
  # ...
  defguard is_set(value) when is_map(value) or is_list(value) or is_tuple(value)
  # ...
end

Now we can start building our custom set_attribute:

# lib/tracing.ex
defmodule Tracing do
  # ...
  def set_attribute(key, value) when is_set(value) do
    set_attributes(key, value) # To be yet defined!
  end

  def set_attribute(key, value) do
    OpenTelemetry.Tracer.set_attribute(key, value)
  end
  # ...
end

It's simple: if the value is a set, we call set_attributes (in plural!), otherwise, we just delegate to OpenTelemety.Tracer.set_attribute/2.

The next piece is where all the transformation happens:

# lib/tracing.ex
defmodule Tracing do
  # ...
  def set_attributes(key, values) do
    key
    |> enumerable_to_attrs(values) # To be yet defined!
    |> OpenTelemetry.Tracer.set_attributes()
  end
  # ...
end

Here, we receive the set of values, convert them, and then call OpenTelemetry.Tracer.set_attributes/1 to do the actual work of adding the attributes to the span.

The data conversion happens in the enumerable_to_attrs/2 function. It works by recursively going into each element of the collection and converting it to the appropriate basic type supported by OpenTelemetry. Adding its code here is beyond the scope of this post, but feel free to check it out on GitHub!

Wrapping Up

In this post, we discussed the basics of tracing and began to explore how we can utilize OpenTelemetry in Elixir. We laid the foundation for an abstraction layer that will simplify the creation and manipulation of spans, making the process seamless and straightforward.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

AppSignal’s Top 5 Elixir Posts in 2023

Roy Tomeij — 2023-12-20T00:00:00+00:00

As the year draws to an end, we're excited to share our top five most-read Elixir articles in 2023!

Season's Greetings ❆ ⛄

Have a wonderful festive break, and we'll see you in the new year!

If you haven’t already, don’t forget to subscribe to our Elixir Alchemy newsletter, so you never miss an upcoming post.

What's New in Elixir 1.16

Roy Tomeij — 2023-12-19T00:00:00+00:00

The Elixir 1.16 release candidate is out now, and it comes with some compelling improvements to diagnostics, documentation, and a few other enhancements that make Elixir an even better choice for developers.

We'll dive into some of these changes and highlight Elixir's continued focus on developer happiness and community building.

Let's get started!

Better Diagnostics for Compiler Errors in Elixir

We'll begin with the improvements to compiler diagnostics provided in Elixir 1.16. Now, your mix compile errors come with detailed code snippets that tell you exactly where your code goes wrong.

You'll benefit from these improvements when you have syntax errors, mismatched delimiter errors, and generic compiler errors.

Syntax Errors

First up, you can see below that syntax errors now come with a pointer to exactly where the error happened. Let's say we have the following code:

defmodule Doctor do
  def show_prescription_codes do
    ["DR", "DS", "amp", &]
  end
end

If you run mix compile, you'll see a syntax error with an associated code snippet, like this:

sophiedebenedetto[ 7:30PM] ~/personal-projects/doctor  💖 mix compile
Compiling 1 file (.ex)

== Compilation error in file lib/doctor.ex ==
** (SyntaxError) invalid syntax found on lib/doctor.ex:48:26:
    error: syntax error before: ']'
    │
 48 │     ["DR", "DS", "amp", &]
    │                          ^
    │
    └─ lib/doctor.ex:48:26
    (elixir 1.16.0-rc.1) lib/kernel/parallel_compiler.ex:428: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/8

Note the ^ pointer to the exact location of the syntax error. With this, it's even easier to quickly spot and correct such errors.

Mismatched Delimiter Errors

Another diagnostic improvement comes with the code snippet representation for MismatchedDelimiterError occurrences. Given the following code with a mismatched delimiter:

defmodule Doctor do
  def show_prescription_codes do
    ["DR", "DS", "amp", "ad")
  end
end

Running mix compile will now show you the following:

sophiedebenedetto[ 7:29PM] ~/personal-projects/doctor  💖 mix compile
Compiling 1 file (.ex)

== Compilation error in file lib/doctor.ex ==
** (MismatchedDelimiterError) mismatched delimiter found on lib/doctor.ex:48:29:
    error: unexpected token: )
    │
 48 │     ["DR", "DS", "amp", "ad")
    │     │                       └ mismatched closing delimiter (expected "]")
    │     └ unclosed delimiter
    │
    └─ lib/doctor.ex:48:29
    (elixir 1.16.0-rc.1) lib/kernel/parallel_compiler.ex:428: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/8

This error includes a display of the code snippet that is causing it, and even highlights the unclosed and mismatched delimiters associated to the MismatchedDelimiterError.

Other Errors

You'll now see similarly detailed code snippets for all kinds of compiler errors, including errors that describe undefined variables. Let's look at an example of that now.

Given the following code:

def list_patients do
  Patient.for_doctor(doctor)
end

Running mix compile will display the following error:

sophiedebenedetto[ 7:36PM] ~/personal-projects/doctor  💖 mix compile
Compiling 1 file (.ex)
    error: undefined variable "doctor"
    │
 53 │     Patient.for_doctor(doctor)
    │                        ^^^^^^
    │
    └─ lib/doctor.ex:53:24: Doctor.list_patients/0


== Compilation error in file lib/doctor.ex ==
** (CompileError) lib/doctor.ex: cannot compile module Doctor (errors have been logged)

These improved diagnostics make it even easier for Elixir developers to understand compilation errors, catch and remedy basic mistakes, and write clean and functioning code. These improvements show Elixir's continued focus on developer experience.

ExDoc Content Matures with Elixir

Some of the most exciting changes coming in Elixir 1.16 are the improvements to official Elixir language docs, offered through ExDoc. The "Getting Started" guide from elixir-lang.org has been incorporated into the official Elixir documentation on Hex docs.

While this may not seem too exciting on the surface, it represents a concerted effort to refine and unify official guidance on the Elixir programming language. The language and the community have matured to such a stage that we can better align on official guides like this and provide the best agreed-upon direction for new developers.

Anti-Patterns

In this same vein, the official Elixir docs now include robust content on Elixir anti-patterns. This content is divided into four separate sections, covering code, design, processes, and metaprogramming.

Once again, this content is made possible by the maturity of the Elixir language. As Elixir has grown up, and as adoption and the community have grown, we're at a point where Elixir developers can align on what good and bad Elixir code looks like.

With this extensive guidance in the official docs, including framing context and code samples, Elixir developers have the resources they need to ship clean code that meets generally agreed-upon standards. It will be even easier for experienced and novice Elixir developers alike to write and maintain clean code in various scenarios.

And that's not it for docs enhancements.

ExDoc Cheatsheets In the Official Elixir Docs

The official docs now include an ExDoc cheatsheet for the first time. ExDoc cheatsheets allow developers to author quick, summary-type guides to modules and libraries.

The official Elixir docs now include an Enum module cheatsheet, and that's just the beginning.

Elixir maintainers are looking for more community members to contribute cheatsheets to the official docs. This is a great way for first-timers to get an Elixir language contribution! Get started by opening an issue that outlines what cheatsheet you'd like to add.

Taken together, these docs improvements all point to the maturity of the Elixir language and community and the continued focus on education and support for Elixir newcomers. The official introduction docs, the anti-pattern docs, and the inclusion of cheatsheets provide clear and accessible guidance to Elixir developers at every level. This kind of alignment and clarity makes Elixir even more accessible to novices. At the same time, it empowers experienced Elixir devs to deliver clean, standardized Elixir code that performs well.

More Enhancements

You might be interested in taking advantage of some of the smaller enhancements from this release in your upgraded Elixir apps. Let's explore them briefly now.

The `String.replace_invalid/2` Function

First up, there is a new String.replace_invalid/2 function that (you guessed it) lets you replace invalid characters in a string. Let's take a look at an example:

iex> String.replace_invalid("asd" <> <<0xFF::8>>)
"asd�"

The default behavior is to replace the invalid character with a �, but you can replace it with any character you like:

iex> String.replace_invalid("asd" <> <<0xFF::8>>, "😵")
"asd😵"

This will come in handy if you find yourself processing strings of text from external sources. Such text may contain invalid characters, and identifying and replacing such characters just got a lot easier.

New Functionality for `Task.yield_many/2`

Next up, let's take a look at the new option you can pass to Task.yield_many. Previously, you could tell Task.yield_many to timeout and stop waiting for tasks if a certain time limit was exceeded. Now, you can provoke that same behavior if a certain number of tasks has been reached. You can do so by providing the limit option.

Let's say you have the following code to spawn ten tasks. Each task will sleep for i number of seconds and then return a message indicating what happened. Then, we yield each task with its result, and iterate over those {task, result} tuples to operate on the result by IO.inspect-ing them:

def MyModule do
  def do_tasks do
    tasks =
      for i <- 1..10 do
        Task.async(fn ->
          Process.sleep(i * 1000)
          "I slept for #{i} seconds"
        end)
      end

    tasks_with_results = Task.yield_many(tasks, timeout: 10000)

    results =
      Enum.map(tasks_with_results, fn {task, res} ->
        # Shut down the tasks that did not reply nor exit
        res || Task.shutdown(task, :brutal_kill)
      end)

    # Here we are matching only on {:ok, value} and
    # ignoring {:exit, _} (crashed tasks) and `nil` (no replies)
    for {:ok, value} <- results do
      IO.inspect(value)
    end
  end
end

This code will continue to yield tasks and their results until all the tasks are done or the timeout is exceeded. Now, you can use the limit: task_num option instead of timeout, like this:

tasks_with_results = Task.yield_many(tasks, limit: 1)

If we execute our function in IEx, you'll see that the code stops waiting for tasks after just one task is yielded:

iex> MyModule.do_tasks()
"I slept for 1 seconds"
["I slept for 1 seconds"]

If the limit is reached before the default timeout (5000), or before all the tasks are done, then Task.yield_many returns immediately without triggering the :on_timeout behaviour. You can dig further into this functionality in the docs.

This new functionality gives you even more fine-grained control over how your program orchestrates async tasks.

The `Logger.levels/0` Function

The last enhancement I'll highlight here is the new Logger.levels/0 function. This function returns the list of all available log levels:

iex> Logger.levels()
[:error, :info, :debug, :emergency, :alert, :critical, :warning, :notice]

This provides a quick reminder of the levels you can use to configure your logger.

Wrap Up

With the 1.16 release, Elixir continues to be a compelling choice for developers looking for:

An excellent developer experience
A welcoming and supportive community
Strong and opinionated development guidelines
Powerful language capabilities to meet the needs of various programming scenarios.

The new developments in diagnostics, documentation, and language functionality continue to make Elixir a pleasure to work with. Better, more detailed diagnostic messages for compiler errors show Elixir's emphasis on developer experience, making it easier than ever before to identify and fix these errors.

Improved docs and the addition of the anti-patterns guide empowers both experienced and newbie Elixir devs to write clean and standardized code. And minor language improvements, like the functions we explored above, show Elixir's continued commitment to language excellence.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Advanced Multi-tenancy for Elixir Applications Using Ecto

Roy Tomeij — 2023-12-05T00:00:00+00:00

Welcome to part two of this series. In the previous tutorial, we learned about multi-tenancy, including different multi-tenancy implementation strategies. We also started building a multi-tenant Phoenix link shortening app and added basic user authentication.

In this final part of the series, we'll build the link resource, associate users to links, and set up the redirect functionality in our app. As we finalize the app build, we'll learn about features (like Ecto custom types) that make Phoenix such a powerful framework for Elixir.

Let's now turn our attention to link shortening.

Adding `Link` Resources

As a reminder, here's how link shortening will happen in our app:

Logged-in user enters a long URL
A random string is generated and associated with the long URL
Whenever this short URL is visited, the app will keep a tally of the number of visits

We'll get started by generating a controller, schema, migration, and views for a Link resource:

mix phx.gen.html Links Link links url:string visits:integer account_id:references:accounts

This should generate most of the boilerplate code we need to work with the Link resource.

You may notice that we haven't included a field for the short random string referencing the long URL. We'll cover this next.

Creating Short URLs Using the Ecto Custom Type in Phoenix

By default, Ecto models have an id field used as a model's primary key. Usually, it's in the form of an integer or, in some cases, a binary ID. In either case, when present in a model, this field is automatically incremented for every additional model created in an app.

A core function of our app is generating short, unique strings to represent long URLs. We could write a custom function to generate such strings, but since we are on a quest to learn Elixir, let's use a more creative approach.

Let's substitute the id primary key in the Link model with a custom Ecto type called HashId.

Using this approach, we'll:

Learn how to create and use Elixir custom types.
Automatically generate unique string hashes to form the short URL field for links. Ecto will manage the process whenever a new link model is created.

First, create a new file to represent this new type:

# lib/ecto/hash_id.ex

defmodule Urlbot.Ecto.HashId do
  @behaviour Ecto.Type
  @hash_id_length 8

  # Called when creating an Ecto.Changeset
  @spec cast(any) :: Map.t
  def cast(value), do: hash_id_format(value)

  # Accepts a value that has been directly placed into the ecto struct after a changeset
  @spec dump(any) :: Map.t
  def dump(value), do: hash_id_format(value)

  # Changes a value from the default type into the HashId type
  @spec load(any) :: Map.t
  def load(value), do: hash_id_format(value)

  # A callback invoked by autogenerate fields
  @spec autogenerate() :: String.t
  def autogenerate, do: generate()

 # The Ecto type that is being converted
  def type, do: :string

  @spec hash_id_format(any) :: Map.t
  def hash_id_format(value) do
    case validate_hash_id(value) do
      true -> {:ok, value}
      _ -> {:error, "'#{value}' is not a string"}
    end
  end

  # Validation of given value to be of type "String"
  def validate_hash_id(string) when is_binary(string), do: true
  def validate_hash_id(_other), do: false

  # The function that generates a HashId
  @spec generate() :: String.t
  def generate do
    @hash_id_length
    |> :crypto.strong_rand_bytes()
    |> Base.url_encode64
    |> binary_part(0, @hash_id_length)
  end
end

Check out the Ecto custom types documentation, which covers the subject in a much more exhaustive way than we could in this tutorial.

For now, what we've just done allows us to use the custom data type HashId when defining a field's data type.

Let's use this new data type to define the short URL field for the Link resource next.

Using the Custom Ecto Type in Phoenix

Open up the Link schema and edit it to reference the new Ecto type we've just defined:

# lib/urlbot/links/link.ex

defmodule Urlbot.Links.Link do
  use Ecto.Schema
  import Ecto.Changeset
  alias Urlbot.Ecto.HashId # Add this line

  @primary_key {:hash, HashId, [autogenerate: true]} # Add this line
  @derive {Phoenix.Param, key: :hash} # Add this line

  ...
end

Before moving on, let's briefly explain the use of @derive in the code above.

In Elixir, a protocol defines an API and its specific implementations. Phoenix.Param is a protocol used to convert data structures into URL parameters. By default, this protocol is used for integers, binaries, atoms, and structs. The default key :id is usually used for structs, but it's possible to define other parameters. In our case, we use the parameter hash and make its implementation derivable to actually use it.

Let's modify the links migration to use this new parameter type:

# priv/repo/migrations/XXXXXX_create_links.exs

defmodule Urlbot.Repo.Migrations.CreateLinks do
  use Ecto.Migration

  def change do
    create table(:links, primary_key: false) do
      add :hash, :string, primary_key: true
      add :url, :string
      add :visits, :integer
      add :account_id, references(:accounts, on_delete: :delete_all)

      timestamps()
    end

    create index(:links, [:account_id])
  end
end

Then run the migration to create the links table:

mix ecto.migrate

We now need to associate Link and Account— but before we do, let's ensure that only an authenticated user can make CRUD operations for Link.

Updating `Link` Routes in Phoenix for Elixir

In the router, we need to create a new pipeline to process routes that only authenticated users can access.

Go ahead and add this pipeline:

# lib/urlbot_web/router.ex

defmodule UrlbotWeb.Router do
  use UrlbotWeb, :router
  use Pow.Phoenix.Router

  ...

  pipeline :protected do
    plug Pow.Plug.RequireAuthenticated, error_handler: Pow.Phoenix.PlugErrorHandler
  end

  ...
end

Then add the scope to handle Link routes:

# lib/urlbot_web/router.ex

defmodule UrlbotWeb.Router do
  use UrlbotWeb, :router
  use Pow.Phoenix.Router

  ...

  pipeline :protected do
    plug Pow.Plug.RequireAuthenticated, error_handler: Pow.Phoenix.PlugErrorHandler
  end

  scope "/", UrlbotWeb do
    pipe_through [:protected, :browser]

    resources "/links", LinkController
  end

  ...
end

Now, if you try to access any of the protected routes, Pow should redirect you to a login page:

Our app build is progressing well. Next, let's associate Link and Account since this forms the basis of tenant separation in our app.

Assigning `Link` to `Account` in Elixir

Take a look at the Link schema below. We want to make sure account_id is included, since this will form the link to Account:

# lib/urlbot/links/link.ex

defmodule Urlbot.Links.Link do
  use Ecto.Schema
  import Ecto.Changeset
  alias Urlbot.Ecto.HashId

  ...

  schema "links" do
    field :url, :string
    field :visits, :integer
    field :account_id, :id

    timestamps()
  end
  ...

end

Next, edit the changeset block, adding account_id to the list of allowed attributes and those that will go through validation:

# lib/urlbot/links/link.ex

defmodule Urlbot.Links.Link do
  use Ecto.Schema
  import Ecto.Changeset
  alias Urlbot.Ecto.HashId

  ...

  def changeset(link, attrs) do
    link
    |> cast(attrs, [:url, :visits, :account_id])
    |> validate_required([:url, :visits, :account_id])
  end

end

Then, edit the Account schema and add a has_many relation as follows:

# lib/urlbot/accounts/account.ex

defmodule Urlbot.Accounts.Account do
  use Ecto.Schema
  import Ecto.Changeset
  alias Urlbot.Users.User
  alias Urlbot.Links.Link

  schema "accounts" do
    field :name, :string

    has_many :users, User
    has_many :links, Link # Add this line

    timestamps()
  end

  ...

end

We also need to edit the Link context to work with Account. Remember, scoping a resource to a user or account is the foundation of any multi-tenant structure.

Edit your file as shown below:

# lib/urlbot/links.ex

defmodule Urlbot.Links do
  ...

  def list_links(account) do
    from(s in Link, where: s.account_id == ^account.id, order_by: [asc: :id])
    |> Repo.all()
  end

  def get_link!(account, id) do
    Repo.get_by!(Link, account_id: account.id, id: id)
  end

  def create_link(account, attrs \\ %{}) do
    Ecto.build_assoc(account, :links)
    |> Link.changeset(attrs)
    |> Repo.insert()
  end

  ...

end

Adding a `Current_Account`

Our edits to the Link context show that most related controller methods use the account variable. This variable needs to be extracted from the currently logged-in user's account_id and passed on to the respective controller action.

So we need to make a custom plug to add to the conn. Create a new file — lib/plugs/set_current_account.ex — and edit its content as shown below:

# lib/plugs/set_current_account.ex

defmodule UrlbotWeb.Plugs.SetCurrentAccount do
  import Plug.Conn
  alias Urlbot.Repo
  alias Urlbot.Users.User

  def init(options), do: options

  def call(conn, _opts) do
    case conn.assigns[:current_user] do
      %User{} = user ->
        %User{account: account} = Repo.preload(user, :account)
        assign(conn, :current_account, account)
      _ ->
        assign(conn, :current_account, nil)
    end
  end
end

Next, let's use this new plug by adding it to the router (specifically to the protected pipeline since user sessions are handled there):

# lib/urlbot_web/router.ex

defmodule UrlbotWeb.Router do
  use UrlbotWeb, :router
  use Pow.Phoenix.Router

  ...

  pipeline :protected do
    plug Pow.Plug.RequireAuthenticated, error_handler: Pow.Phoenix.PlugErrorHandler
    plug UrlbotWeb.Plugs.SetCurrentAccount # Add this line
  end

  ...

  scope "/", UrlbotWeb do
    pipe_through [:protected, :browser]

    resources "/links", LinkController
  end

  ...

end

Then, edit the Link controller to pass the current_account in every action where it's required, like so:

# lib/urlbot_web/controllers/link_controller.ex

defmodule UrlbotWeb.LinkController do
  use UrlbotWeb, :controller

  alias Urlbot.Links
  alias Urlbot.Links.Link

  def index(conn, _params) do
    current_account = conn.assigns.current_account  # Add this line
    links = Links.list_links(current_account)
    render(conn, :index, links: links)
  end
  def create(conn, %{"link" => link_params}) do
    current_account = conn.assigns.current_account # Add this line
    case Links.create_link(current_account, link_params) do
      {:ok, link} ->
        conn
        |> put_flash(:info, "Link created successfully.")
        |> redirect(to: ~p"/links")

      {:error, %Ecto.Changeset{} = changeset} ->
        render(conn, :new, changeset: changeset)
    end
  end

  def edit(conn, %{"id" => id}) do
    current_account = conn.assigns.current_account  # Add this line
    link = Links.get_link!(current_account, id)
    changeset = Links.change_link(link)
    render(conn, :edit, link: link, changeset: changeset)
  end

  def update(conn, %{"id" => id, "link" => link_params}) do
    current_account = conn.assigns.current_account  # Add this line
    link = Links.get_link!(current_account, id)

    case Links.update_link(link, link_params) do
      {:ok, link} ->
        conn
        |> put_flash(:info, "Link updated successfully.")
        |> redirect(to: ~p"/links/#{link}")

      {:error, %Ecto.Changeset{} = changeset} ->
        render(conn, :edit, link: link, changeset: changeset)
    end
  end

  def delete(conn, %{"id" => id}) do
    current_account = conn.assigns.current_account  # Add this line
    link = Links.get_link!(current_account, id)
    {:ok, _link} = Links.delete_link(link)

    conn
    |> put_flash(:info, "Link deleted successfully.")
    |> redirect(to: ~p"/links")
  end

end

That's it! Now, any created Link will be associated with a currently logged-in user's account.

At this point, we have everything we need for users to create links that are properly scoped to their respective accounts. You can see this in action when you consider the list index action:


defmodule UrlbotWeb.LinkController do
  use UrlbotWeb, :controller

  alias Urlbot.Links
  alias Urlbot.Links.Link

  def index(conn, _params) do
    current_account = conn.assigns.current_account
    links = Links.list_links(current_account)
    render(conn, :index, links: links)
  end

  ...
end

Here, a user should only see links that are associated with them.

For example, this is a list index view for one user:

And this is a view for a different user (notice the logged-in user's email and the different link URLs):

Next, let's build the link redirection feature.

Redirecting Links and Incrementing Link Views in Phoenix

Let's begin by adding a new controller to handle the redirect. This controller will have one show action:

# lib/urlbot_web/controllers/redirect_controller.ex

defmodule UrlbotWeb.RedirectController do
  use UrlbotWeb, :controller

  alias Urlbot.Links

  def show(conn, %{"id" => id}) do
    short_url = Links.get_short_url_link!(id)
    Links.increment_visits(short_url)
    redirect(conn, external: short_url.url)
  end

end

We also need to modify the Link context with a custom get function that does not reference the current user:

# lib/urlbot/links.ex

defmodule Urlbot.Links do
  ...

  def get_short_url_link!(id) do
    Repo.get!(Link, id: id)
  end

  ...

end

Next, modify the router accordingly:

# lib/urlbot_web/router.ex

defmodule UrlbotWeb.Router do
  use UrlbotWeb, :router
  use Pow.Phoenix.Router

  ...

  scope "/", UrlbotWeb do
    pipe_through :browser

    get "/", PageController, :home
    get "/links/:id", RedirectController, :show # Add this line
  end

  ...

end

Finally, to handle the link view visits, we'll add a custom function to the Link context:

# lib/urlbot/links.ex

defmodule Urlbot.Links do
  ...

  def increment_visits(%Link{} = link) do
    from(s in Link, where: s.id == ^link.hash, update: [inc: [visits: 1]])
    |> Repo.update_all([])
  end

end

That's it! Now, when we visit a link like http://localhost:4000/links/rMbzTz3o, this should redirect to the original long URL and increment the link's view counter.

Wrapping Up

This two-part series has taken you on a journey to build a multi-tenant Elixir app using the Phoenix web framework. You've learned how to implement a simple tenant separation system using foreign keys and related models within a shared database and shared schema.

Although we've tried to be as exhaustive as possible, we couldn't possibly capture the whole app-building process, including testing, deployment options, or even the use of the amazing LiveView.

All the same, we hope this tutorial provides a foundation for you to build your very own Elixir app that helps solve serious problems for your users.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Setting Up a Multi-tenant Phoenix App for Elixir

Roy Tomeij — 2023-11-21T00:00:00+00:00

Apps built with Elixir can support massive scalability, real-time interactivity, great fault tolerance, and the language's syntax is actually a joy to use. Elixir is a natural fit for applications such as chat apps, data dashboard apps, and anything needed to support a large userbase.

In this article, we'll use Elixir — specifically, the web framework Phoenix — to build a multi-tenant link shortening app.

Let's get started!

What's Needed for Our Phoenix Project Built with Elixir

With the growing popularity of social media, there's been an increase in the use of URL shortening services. As a budding entrepreneur, you believe there's a market for yet another URL shortening service. And since such an app lives in between a generated short URL and the long version, it's necessary that the app be fast and scalable.

These features and more are already built into Phoenix, which makes it a great choice for this project, enabling you to stand out in the market.

Takeaways

After this tutorial, you will:

Understand the internal workings of an Elixir/Phoenix app (for example, how to create and use a custom Ecto type, how to handle authentication, and more).
Understand the concept of multi-tenancy, including the different strategies available.
Build a multi-tenant link shortening Elixir/Phoenix app that can be extended into something more advanced with real-world use.

To follow along with the tutorial, let's set up a few things.

Prerequisites

Elixir installed on your development machine. If you don't have it installed, follow this guide. For this tutorial, we're using Elixir version 1.14.0.
Since we'll be using Phoenix for the web app build, you'll also need to have it installed on your local development machine. We're using Phoenix version 1.7.2 in this tutorial.
PostgreSQL - Ensure you have PostgreSQL installed, as it's the default database used by Elixir/Phoenix apps.

With that in place, let's see what kind of app we'll be building next.

An Overview of the Phoenix App We'll Build

The app that we'll build is a simple multi-tenant URL shortening app with the following features:

User authentication - users can register and log in to the application.
Multi-tenant separation - users can create shortened links within their own separate accounts.
URL shortening - users can input normal links and generate shortened versions that redirect back to the original link when clicked.
View metrics - users can see how many times a link was clicked.

Implementing these features should add to your experience in working with Elixir and provide a good foundation to build even more complex Elixir apps.

Next, let's understand what multi-tenancy is and how it works.

An Overview of Multi-tenancy

At the most basic level, a multi-tenant app is one where the same application and database serve several tenants, with each client's data kept separate from that of other clients. There are thousands of real-world multi-tenant app examples, including AppSignal, Gmail, and others.

In our app, a user will be able to register and create shortened links that are associated with their account. Other users will also be able to sign up and create shortened links associated to them, but users won't be able to see other users' links. This is a simplified but very good example of multi-tenancy in action.

In the next section, we'll outline the different strategies a developer can use when building a multi-tenant app. It's important to point out that these strategies aren't exclusive to Elixir. Rather, think of them as universal building blocks for developing multi-tenant apps regardless of the programming language used.

Multi-tenancy Strategies

Several multi-tenancy strategies are available, including:

Separate databases - Here, each tenant gets their own database for complete isolation of each client's data. The beauty of this approach is that it's very scalable and great for applications where data security and isolation is key: for example, patient medical records, financial records, and other similar use cases. As you can imagine, one of the biggest disadvantages with this approach is how complex and costly it is to build and maintain apps using this strategy.
A shared database with separate schema - In this strategy, each client gets a separate schema within a shared database. This approach allows for a scalable system that is not too complex to build and maintain. That said, having separate schemas for each client is not something you can easily use to handle massive scale.
A shared database with shared schema - Here, all tenants share a common database and schema. It is a great choice for low-to-medium traffic apps and offers a convenient way to get started with multi-tenancy. Many SaaS startups are often built using this strategy. The biggest disadvantage of this strategy is that it's not built for scale or speed.
Hybrid strategy - A not-so-common approach where you have both the shared database and shared schema for some groups of users (say, the ones that pay the least in a SaaS app), and a common database with a separate schema for premium customers. This approach offers some level of scaling but is very complex to build and maintain.
Containerization - Similar to the separate databases approach, here, each tenant is provided with a completely separate and isolated app container. The obvious advantages are speed and scalability, but this is complex to build and maintain.

You now have a good overview of what strategy to use when you build your own Elixir app next time.

For this app project, we'll be using the shared database with shared schema approach and an app stack described in the next section.

Our Elixir Application Stack

To build our app project, we will use the up-and-coming Elixir stack called "PETAL", short for:

P - Phoenix
E - Elixir
T - Tailwind CSS
A - Alpine JS
L - Liveview - LiveView is an Elixir library that you include as a dependency in a Phoenix app to handle interactivity and other real-time flows characteristic of single-page applications (SPAs).

Since our goals are simple, employing the full PETAL framework in this case is overkill. Instead, we'll use the simple stack you get when you generate a new Phoenix web application:

Phoenix
PostgreSQL
HTML and Tailwind CSS

We are now ready to jump into the build, but before we do, it's necessary that we understand how link shortening actually works. This will form the basis for the steps we'll take during the build.

How Link Shortening Works

Link shortening is actually very simple. The link shortening app lies between a long URL and a generated short URL. When a visitor hits a short URL, their request is received by the app and a quick database search is done to find the matching long URL. Once that long URL is found, the visitor is redirected to it, and the link visits counter updates.

Obviously, this is a very simplified outline, but it will suffice for now. Next, let's start off our build.

Generating a New Phoenix Application

Begin by generating a new Phoenix application using the following mix command:

mix phx.new urlbot

Then open your project directory in a text editor and edit the dev.exs file with your development environment's database settings:

# config/dev.exs

import Config

config :shorten_my_links_app, ShortenMyLinksApp.Repo,
  username: "DATABASE USERNAME", # Edit this line
  password: "DATABASE PASSWORD", # Edit this line
  hostname: "localhost",
  database: "DATABASE NAME", # Edit this line
  stacktrace: true,
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

  # ...

Create the database for the application:

mix ecto.create

Once that is done, run mix phx.server in the terminal to compile the application and spin up an instance of the compiled app on localhost:4000, where you can see the default Phoenix home page:

Next up, let's set up the multi-tenant structure and user authentication.

Setting Up Multi-tenancy for Elixir

The tenant structure needs to be implemented as early as possible into a project build, since doing it later can result in all sorts of challenges. To put the shared database with shared schema multi-tenant strategy that we chose into actuality, we'll make use of foreign keys and related models.

Specifically, we'll have two main models: User and Account (this can also be called Organization, Team, or even Company); a User belongs to the Account model.

We'll also have a third Link model which will belong to the Account model. This way, the User model can be used exclusively for authentication purposes, while resource ownership will be handled by Account.

This structure is represented in the diagram below:

With this structure in place, all resources created, updated, or deleted in the app can be scoped to a specific Account and User. And just like that, we can achieve our goal of having separate tenant resources in a shared database with shared schema setup.

An additional benefit to using this structure is that you can easily expand it to invite other users to an account as "teammates" and assign them different roles. However, we won't cover this feature in this tutorial.

Let's begin by generating the Account context.

Generating an `Account` Context

Generate the Account context, schema, and migration file with the command below:

mix phx.gen.context Accounts Account accounts name:string

Then run the migration with:

mix ecto.migrate

Next, let's move on to the User.

Building a `User` Context and Authentication with Elixir

The User context will be used for authentication purposes. Instead of generating the User context and trying to integrate it into an authentication flow from scratch, Elixir gives us several libraries we can use, including phx.gen.auth, Coherence, and Pow, a modular and extendable authentication library for Phoenix apps.

We first add Pow to the project's dependencies:

# mix.exs

defp deps do
  [
    # ...
    {:pow, "~> 1.0.30"}
  ]
end

And fetch it:

mix deps.get

Then finish by installing it:

mix pow.install

With that, we get a User context, schema, and migration file. User authentication routes are also appended to the router.

At this point, we could run the migration, but there are a couple of changes we should make to our generated user files first to ensure they are properly related.

Adding the `User` and `Account` Relationship

Begin by adding a belongs_to association to User:

# lib/shorten_my_links_app/users/user.ex

defmodule Urlbot.Users.User do
  use Ecto.Schema
  use Pow.Ecto.Schema

  schema "users" do
    pow_user_fields()

    belongs_to :account, Urlbot.Accounts.Account # Add this line

    timestamps()
  end
end

Let's also modify Account to include the has_many relationship:

# lib/urlbot/accounts/account.ex

defmodule Urlbot.Accounts.Account do
  use Ecto.Schema
  import Ecto.Changeset

  schema "accounts" do
    field :name, :string

    has_many :users, Urlbot.Users.User # Add this line

    timestamps()
  end

  ...
end

Next, let's modify the users migration to add the account_id field as a foreign key. We also need to indicate that a user will be deleted whenever their related account is deleted:

# priv/repo/migrations/XXXXXXXXXX_create_users.exs

defmodule Urlbot.Repo.Migrations.CreateUsers do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :email, :string, null: false
      add :password_hash, :string

      # Add this line
      add :account_id, references(:accounts, on_delete: :delete_all), null: false

      timestamps()
    end

    create index(:users, [:account_id]) # Add this line
    create unique_index(:users, [:email])
  end
end

Finally, it's nice to have an Account automatically created the first time a user registers on the app.

Creating an `Account` on User Registration

We need a way to capture the one attribute of the Acccount model (the account_name in the user registration form) and pass this attribute to a modified user creation process which will create a related Account for us.

That sounds like a lot, but let's go step-by-step.

First, add the account_name attribute to the user registration form. Since we are working with Pow which comes with pre-built view templates, we need to generate the templates by running the command:

mix pow.phoenix.gen.templates

This will generate Pow's view templates, but we are only interested in the user registration view for now. Edit it by adding the account_name field:

# lib/urlbot_web/controllers/pow/registration_html/new.html.heex

<div class="mx-auto max-w-sm">
  ...

  <.simple_form :let={f} for={@changeset} as={:user} action={@action} phx-update="ignore">
    ...

    <!-- Add this line -->
    <.input field={f[:account_name]} type="text" label="Account name" />

    ...
  </.simple_form>
</div>

By the way, instead of adding the account_name like this, we can use a nested form for working with Account from User (but this method should work just as well).

Next, we want to add account_name as a virtual attribute in the User schema. The reason it's a virtual attribute is simply because it's not an attribute that is built into the User schema, but we still need to use it in there. You can read more on virtual attributes.

# lib/urlbot/users/user.ex

defmodule Urlbot.Users.User do
  use Ecto.Schema
  use Pow.Ecto.Schema

  schema "users" do
    pow_user_fields()

    # Add this line
    field :account_name, :string, virtual: true

    belongs_to :account, Urlbot.Accounts.Account

    timestamps()
  end
end

At this point, you can run mix ecto.migrate to create the user table.

Next, we want to make sure that the virtual attribute we've just added is passed on to the User changeset, which we'll add since it's not included by default when we run the Pow generator.

We'll also add a custom private function to create an account at the same time a user is created:

# lib/shorten_my_links_app/users/user.ex

defmodule Urlbot.Users.User do
  use Ecto.Schema
  use Pow.Ecto.Schema

  # Add this line since we'll be using Ecto's changeset in this schema
  import Ecto.Changeset

  # Add this line to reference Account since it's used in the private function
  alias Urlbot.Accounts

  ...

  # Also add this block of code to add a changeset
  def changeset(user, attrs) do
    user
    |> pow_changeset(attrs)
    |> cast(attrs, [:account_name])
    |> validate_required([:account_name])
    |> create_user_account(user)
    |> assoc_constraint(:account)
  end

  # Add the custom private function
  defp create_user_account(%{valid?: true,  changes: %{account_name: account_name}} = changeset, %{account_id: nil} = _user) do
    with {:ok, account} <- Accounts.create_account(%{name: account_name}) do
      put_assoc(changeset, :account, account)
    else
      _ -> changeset
    end
  end

  defp create_user_account(changeset, _), do: changeset

end

Let's break down what's going on. First, we define a changeset block passing in a user and the user attributes. Then we cast the virtual attribute we added, followed by a validation rule to make sure it's present. We then call the private function create_user_account (defined just below the changeset block) and finalize with an assoc_constraint which checks that the associated field, account_id, exists.

With all that done, whenever a new user registers, a connected account will be automatically created. Any link resources created by the logged-in user will always be scoped to them. For example, an index view under /links will only display links created by the user, and not list other users' links, even if they are available. This resource separation at the account or user level is the foundation of a multi-tenant structure.

Up Next: Adding `Link` Resources and Custom Ecto Types

We've explored how multi-tenancy works and discussed a few multi-tenancy strategies. We created a Phoenix app and set up multi-tenancy. Finally, we added accounts, users and authentication, and associated users and accounts.

In the next and final part of this series, we'll look at:

Generating the link resource
Using Ecto custom types
Generating a current_account
Assigning a current_account to links
Redirecting links and updating the views counter

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Validating Data in Elixir: Using Ecto and NimbleOptions

Roy Tomeij — 2023-11-07T00:00:00+00:00

In the previous part of this series about validating data at the boundary of an Elixir application, we covered a few general programming tactics to try and reject invalid and unexpected data in our software.

Continuing with that subject, we'll now explore how two libraries, namely Ecto and NimbleOptions, can further assist us.

Let's get started!

Using Ecto

As we've seen previously, Elixir provides many native techniques to help us guarantee data quality in our systems.

But it's also possible to leverage Ecto to cast, validate, and prune data even if there's no database interaction.

Schemaless Changesets in Ecto

Schemaless changesets (basically Ecto changesets that aren't tied to a database table) are a convenient way to create data structures. They also prevent bad data from making its way into structs (this can happen when bad data makes it past constructors or is added directly to a struct).

This approach is typically helpful when dealing with untrusted input (e.g., from an API request or as part of a module's public API). We can expose a function that will accept a plain map and, after proper vetting, will yield a struct. Other functions in the same module can then accept such a struct instance (rather than a map) to indicate that the data has been vetted and is safe to consume.

Here's what that approach can look like:

# in the Account module

defstruct [:name, suspended: false]

def from_params(%{} = params) do
  data  = %{}
  types = %{name: :string, suspended: :boolean}

  changeset =
    {data, types}
    |> Ecto.Changeset.cast(params, Map.keys(types))
    |> Ecto.Changeset.validate_required(...)
    |> Ecto.Changeset.validate_length(...)

  case apply_action(changeset, :insert) do
    {:ok, data} -> {:ok, struct(__MODULE__, data)}
    {:error, _} = error -> error
  end
end

Note that we're applying an :insert action, so we could even use our function's error to display directly in a Phoenix form: those require an :insert or :update action to render possible errors with the form's data.

Applying This To Phoenix Forms

Indeed, Phoenix forms inspect the action to determine whether error hints should be displayed: if no action is set, no errors will be rendered in the form. This is useful if an empty changeset is being used to render a Phoenix form: the changeset is invalid, but we don't want to berate the user with errors when they haven't (yet) made any actual errors. But this also means that if you want the validation errors resulting from from_params to show up in a Phoenix form, you need to set the :action value yourself, which is what we're accomplishing with our call to apply_action.

Of course, if you don't plan on using this with a Phoenix-rendered form, it won't be needed and can be safely skipped.

Creating a new validated account is now a simple matter of Account.from_params(%{name: "ACME", suspended: false}).

And since the types are dynamic, they could also be passed in as arguments if your domain requires it.

Embedded Schemas in Ecto

Let's say you already have an Account struct that isn't persisted to a database but still want to use Ecto to validate the data. In that case, you can transition to using an embedded schema rather than a basic struct. The trade-off between a schemaless changeset and an embedded schema is that the latter provides a bit more convenience at the expense of flexibility.

Namely, embedded schemas require you to have a struct within which to define the schema. Schemaless changesets don't have that requirement since, as their name implies, they don't need a schema definition.

Also, the datatypes for attributes can be dynamic in the schemaless changeset case (and provided as an argument to a function call, for example). In contrast, embedded schema types cannot vary from their defined value. Here's the above example rewritten to use an embedded schema:

# in the Account module

@primary_key false
embedded_schema do
  field :name, :string
  field :suspended, :boolean
end

def from_params(%{} = params) do
  changeset =
    %__MODULE__{}
    |> Ecto.Changeset.cast(params, [:name, :suspended])
    |> Ecto.Changeset.validate_required(...)
    |> Ecto.Changeset.validate_length(...)

  case apply_action(changeset, :insert) do
    {:ok, data} -> {:ok, struct(__MODULE__, data)}
    {:error, _} = error -> error
  end
end

If you'd like to dig deeper into how Ecto can also help you with your non-persisted data, I highly recommend reading Ecto's Data mapping and validation guide.

Validating Options

There are many circumstances where data is passed in as keyword lists (e.g., options given to OTP modules such as GenServers). These values need to be validated for conformity, but we also want that validation to be easy to understand and communicate. Enter NimbleOptions!

The NimbleOptions Library for Elixir

NimbleOptions is a great tool for validating options, as it is a lightweight library that verifies keyword lists and returns errors on invalid data. Let's see how it can be used!

Say we want to email suspended Account records. The email template to use will be different if we email a corporate or personal account. Additionally, you want to specify which values to pass into the template.

We'll want to adapt the signature: corporate accounts, for example, should have an account manager's name attached, while private accounts can simply have a generic signature. We'll also specify which URL to include as the call to action:

def email(%Account{} = account, opts) when is_list(opts) do
  template = Keyword.fetch!(opts, :template)
  values = Keyword.get(opts, :values, [])
  account_url = Keyword.get(values, :landing_url)
  signature = Keyword.get(values, :signature, "Yours truly,\nACME.com")

  # email generation and sending goes here

  :ok
end

This is a great start: we're enforcing the :template to be provided and the program will crash if the template isn't given. This keeps out bad data, but isn't a great experience for callers: if they make a mistake (even as simple as a typo on the template name!) everything is going to crash instead of just generating an error they can handle.

We could, of course, do something like this:

def email(%Account{} = account, opts) when is_list(opts) do
  with template when template in ~w(personal corporate)a <- Keyword.get(opts, :template) do
    values = Keyword.get(opts, :values, [])
    landing_url = %URI{} = Keyword.get(values, :landing_url, URI.parse("https://www.example.com/sign_in"))
    signature = Keyword.get(values, :signature, "Yours truly,\nACME.com")

    # email generation and sending goes here

    :ok
  else
    nil -> {:error, :missing_tempate}
    other when is_atom(other) -> {:error, :invalid_template_value}
    _ ->  {:error, :invalid_template_type}
  end
end

But that doesn't scale very well once the number of options grows: it gets pretty difficult to see what options are permitted, what their expected type is, and so on. Not to mention, this is going to get repetitive really fast.

Bringing In NimbleOptions

Let's try out NimbleOptions. We need to specify a schema that the options are expected to conform to:

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        type: {:struct, URI},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

def email(%Account{} = account, opts) when is_list(opts) do
  with {:ok, validated_options} <- NimbleOptions.validate(opts, @email_opts_schema) do
    template = Keyword.fetch!(validated_options, :template)
    landing_url = validated_options[:values][:landing_url]
    signature = validated_options[:values][:landing_url]

    # email generation and sending goes here

    {:ok, validated_options}
  end
end

Do note we've specified default: [] in the values configuration: that way, an empty list will be used as the default, enabling the cascading of default child values such as landing_url to be filled in. Without it, a default landing_url won't be filled in if a values keyword isn't present at all.

Here it is in action:

iex> email(an_account, template: :personal)

{:ok,
 [
   values: [
     signature: "Yours truly,\nACME.com",
     landing_url: %URI{
       authority: "www.example.com",
       fragment: nil,
       host: "www.example.com",
       path: "/sign_in",
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     }
   ],
   template: :personal
 ]}

iex> email(an_account, template: :personal, values: [signature: "With love,\nBob"])

{:ok,
 [
   template: :personal,
   values: [
     landing_url: %URI{
       authority: "www.example.com",
       fragment: nil,
       host: "www.example.com",
       path: "/sign_in",
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     },
     signature: "With love,\nBob"
   ]
 ]}

As we can see, default values are being filled in when required, and they won't overwrite any provided values.

Validation also works out of the box:

iex> email(an_account, template: :boom)

{:error,
 %NimbleOptions.ValidationError{
   key: :template,
   keys_path: [],
   message: "invalid value for :template option: expected one of [:personal, :corporate], got: :boom",
   value: :boom
 }}

It will also verify that we're not passing in unexpected option keys, such as typos:

iex> email(an_account, template: :personal, values: [singature: "With love,\nBob"])

{:error,
 %NimbleOptions.ValidationError{
   key: [:singature],
   keys_path: [:values],
   message: "unknown options [:singature], valid options are: [:landing_url, :signature]",
   value: nil
 }}

Allowing Strings for `landing_page`

While having the landing_page as a URI ensures it at least looks right, having to provide it as a string is a bit of a pain. It would be nice to provide the option as a string, but here's how that currently behaves:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:error,
 %NimbleOptions.ValidationError{
   key: :landing_url,
   keys_path: [:values],
   message: "invalid value for :landing_url option: expected URI, got: \"https://my.app\"",
   value: "https://my.app"
 }}

Luckily, there's an easy fix — we just have to tweak the type definition:

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        # improved type to accept strings
        type: {:or, [{:struct, URI}, :string]},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

Strings are now accepted for the landing_page value:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:ok,
 [
   template: :personal,
   values: [signature: "Yours truly,\nACME.com", landing_url: "https://my.app"]
 ]}

While a step in the right direction, there are now two minor issues:

Our code has to handle both string and URI data types (or we need to manually convert one to another after validation).
Leaving the value as a string doesn't indicate anything. In particular, there's no indication that the value is a valid URI "safe" to process.

Elixir's NimbleOptions To the Rescue!

Once again, NimbleOptions comes to our rescue: we can use a :custom data type that specifies a parser function. In our case, that parser function can simply be URI.new/1, as it already conforms to the expectation set by :custom (namely, returning {:ok, value} or {:error, message}):

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        # also accept strings, but parse them in %URI{}
        type: {:or, [{:struct, URI}, {:custom, URI, :new, []}]},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

Check it out:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:ok,
 [
   template: :personal,
   values: [
     signature: "Yours truly,\nACME.com",
     landing_url: %URI{
       authority: nil,
       fragment: nil,
       host: "my.app",
       path: nil,
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     }
   ]
 ]}

We've now got the best of both worlds: convenient argument types and a standardized representation post-validation!

Wrapping Up

I hope you've enjoyed these two blog posts covering how you can keep bad data out of your Elixir applications while also making your code more expressive.

As we've seen, there are both "plain vanilla" Elixir techniques as well as support you can get from libraries like NimbleOptions to help us reject invalid data from being processed in our code.

By introducing these approaches to our modules, we'll provide more immediate feedback to callers and also make our programs more resilient when faced with unexpected data.

Hopefully, you've now got a few more tools in your belt to try out. Enjoy!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How To Use Zig for Elixir NIFs

Roy Tomeij — 2023-10-24T00:00:00+00:00

Elixir excels at building scalable and maintainable applications. However, sometimes Elixir is not the best language to tackle specific tasks, and it can fall short in some areas, like direct system interaction.

Fortunately, Elixir offers NIFs (Native Implemented Functions): a path for integrating with other languages to improve these gaps. NIFs in Elixir act as an interface to call functions written in native languages like C or Rust, enabling developers to optimize performance-critical sections of their code.

NIFs can be written in many languages like Rust, Python, and Zig. For this tutorial, we will use Zig, due to its simplicity, speed, and safety.

Let's get going!

What We'll Cover

We'll cover the following topics in this tutorial:

Setting up Zig for Elixir NIF development.
Understanding NIFs in Elixir.
Writing a simple NIF in Zig.
Integrating the NIF with an Elixir application.

What Is Zig?

Zig is a general-purpose programming language designed for robustness, optimality, and maintainability.

It is known to provide excellent debugging and error-handling capabilities, making it suitable for various applications (including systems programming, embedded systems, and performance-critical applications).

Setting up Zig for Elixir Development

Let's now look at how we can set up Zig for Elixir NIF development.

Installing Zig

To get started with Zig, download the latest version of the Zig compiler from the official Zig website. Follow the installation instructions for your specific operating system.

Configuring Zig for Elixir NIF Development

To configure Zig for Elixir NIF development, you'll need to include the necessary libraries and dependencies.

Elixir NIFs rely on the Erlang NIF API provided by the erl_nif.h header file. Ensure you have the Erlang development package installed, and include the appropriate path to erl_nif.h in your build script or Zig build file.

Just add the following line to your src/main.zig file:

const erl = @cImport({
    @cInclude("erl_nif.h");
});

Importing this library will allow us to use the Erlang NIF API in our Zig code. It contains the native functions that allow our Zig code to interact with the Erlang VM. Later in this tutorial, we will see how to use this library to create our NIF.

Verifying the Setup

To verify that Zig is correctly set up for Elixir NIF development, we will create a simple "Hello, World!" project in Zig.

Go ahead and run the following commands:

mkdir hello-world
cd hello-world
zig init-exe

If zig has been installed correctly, you should see the following output:

info: Created build.zig
info: Created src/main.zig
info: Next, try `zig build --help` or `zig build run`

Next, we'll run zig build run on our terminal to ensure we can compile and execute a Zig program. If successful, you should see the following output:

All your codebase are belong to us.
Run `zig build test` to run the tests.

Now that we are able to compile and run Zig code, we can move forward.

Understanding NIFs in Elixir

Why Use NIFs in Elixir?

NIFs can be called from Elixir code, providing a performance boost for computationally intensive tasks. While Elixir excels at concurrency and fault tolerance, there might be better choices for performance-sensitive tasks.

By writing NIFs in a lower-level language like Zig, you can leverage the language's speed and efficiency without sacrificing Elixir's maintainability and concurrency features.

Advantages of Using Zig for NIF Development

Zig offers several advantages for NIF development:

Performance: Zig compiles to efficient native code, providing significant performance improvements over interpreted languages like Elixir.
Safety: Zig has strong static typing, compile-time checks, and a focus on error handling, reducing the likelihood of runtime errors.
Simplicity: Zig's straightforward syntax and semantics make it easy to learn and use, especially for developers familiar with C or C++. Zig's standard library is also very comprehensive, providing a wide range of functionality for common tasks.

Writing our First NIF

Let's start by setting up our Elixir project. Run the following command:

mix new example_nif
cd example_nif

This will create a basic Elixir project called ExampleNif. Next, let's initialize the Zig project inside our Elixir project with the following command:

zig init-lib

This will initialize a library inside our project that will create the following files:

build.zig - which contains the code for compiling and building our zig library.
src/main.zig - this is the main code of our library and will contain the logic that we will implement.

Writing a Simple NIF in Zig

Our first step when working with Zig is to update the build.zig file with the following code:

const std = @import("std");
const Pkg = std.build.Pkg;

pub fn build(b: *std.build.Builder) void {
    const mode = b.standardReleaseOptions();

    const nif_step = b.step("example_lib", "Compiles erlang library");
    const example_lib = b.addSharedLibrary("example_nif", "./src/main.zig", .unversioned);
    example_lib.setBuildMode(mode);
    example_lib.setOutputDir("build");
    example_lib.addIncludePath("/usr/lib/erlang/usr/include/");

    example_lib.install();
    example_lib.linkLibC();
    nif_step.dependOn(&example_lib.step);
}

This will add the Erlang header path to the build environment and also link with the C library.

Next, we will update the src/main.zig code. For this example, we'll create a simple NIF that takes two integers and returns their sum. This function will be written in Zig and called from Elixir code.

const std = @import("std");
const erl = @cImport({
    @cInclude("erl_nif.h");
});

export fn nif_add(
    env: ?*erl.ErlNifEnv,
    argc: c_int,
    argv: [*c]const erl.ERL_NIF_TERM,
) erl.ERL_NIF_TERM {
    var a: i32 = undefined;
    var b: i32 = undefined;

    if ((argc != 2))
    {
        return erl.enif_make_badarg(env);
    }

    _ = erl.enif_get_int(env, argv[0], &a);
    _ = erl.enif_get_int(env, argv[1], &b);

    const result = a + b;
    return erl.enif_make_int(env, result);
}

const func_count = 1;

var funcs = [func_count]erl.ErlNifFunc{
    erl.ErlNifFunc{
        .name = "nif_add",
        .arity = 2,
        .fptr = nif_add,
        .flags = 0,
    },
};

var entry = erl.ErlNifEntry{
    .major = erl.ERL_NIF_MAJOR_VERSION,
    .minor = erl.ERL_NIF_MINOR_VERSION,
    .name = "Elixir.ExampleNif",
    .num_of_funcs = func_count,
    .funcs = &funcs,
    .load = null,
    .reload = null,
    .upgrade = null,
    .unload = null,
    .vm_variant = "beam.vanilla",
    .options = 1,
    .sizeof_ErlNifResourceTypeInit = @sizeOf(erl.ErlNifResourceTypeInit),
    .min_erts = "erts-10.4",
};

export fn nif_init() *erl.ErlNifEntry {
    return &entry;
}

While this code may look intimidating, it's actually quite simple. Let's break it down:

We are first importing the erl_nif.h header file, which contains the definitions for the Erlang NIF API and the std library.
Next, we define our first and only library function, nim_add, which takes three arguments:
- env is a pointer to an ErlNifEnv struct, which contains the environment for the NIF. This struct is used to allocate memory, create Erlang terms, and more.
- argc is the number of arguments passed to the NIF.
- argv is an array of Erlang terms (the arguments passed to the NIF).
- The function code itself is a very simple and naive implementation that adds two integers.
After we have defined all our functions, we define a constant func_count (the number of functions in the NIF).
We then define an array of ErlNifFunc structs, containing the name, arity, and function pointer for each function in the NIF.
Finally, we define an ErlNifEntry struct, which contains the NIF's version, name, and functions. This struct is used to register the NIF with the Erlang VM.

It is important to highlight that the num_of_funcs field in the ErlNifEntry struct must match the number of functions in the funcs array. The name field must also match the module name and function in the NIF_MOD_FUNCS of the build.zig file.

Compiling the Zig Code Into a NIF

To compile the Zig code into a NIF, use the zig build-lib command, specifying the appropriate target and output file. For example, on a Linux system:

zig build example_lib

This command will produce a shared library file: libexample_nif.so in the build/ directory.

Integrating the NIF with Elixir

To use the NIF in Elixir, we need to follow several steps:

Load the shared library containing the NIF.
Define a fallback function to handle cases where the NIF is not loaded.
Call the NIF function from our Elixir code.

Loading the Shared Library

When using NIFs, Elixir needs to load the shared library containing the native code. To do this, we use the :erlang.load_nif/2 function, which takes two arguments:

The relative path to the shared library (.so on Linux, .dll on Windows, or .dylib on macOS).
An optional integer value (usually 0) can be used for versioning purposes.

In our Elixir module, we'll define a load_nif/0 private function that handles loading the shared library. We'll also use the @on_load module attribute to specify that load_nif/0 should be called when the module is loaded.

Defining a Fallback Function

It's important to provide a fallback function that will be called if the NIF fails to load. This function should have the same name and arity as the NIF function and return an error tuple such as {:error, reason} or call the :erlang.nif_error/1 BIF (Built-In Function) with an appropriate error reason.

In our example, we define an add/2 fallback function that calls :erlang.nif_error(:nif_not_loaded).

Calling the NIF Function from Elixir Code

Once the shared library is loaded and the NIF function is linked to the Elixir module, you can call the NIF function just like any other Elixir function. In our example, we call ExampleNif.add/2 to perform the addition using the NIF we wrote in Zig.

By following these steps, you can seamlessly integrate NIFs written in Zig into your Elixir applications, taking advantage of both the performance benefits of Zig and the maintainability and concurrency features of Elixir.

Writing Elixir Code to Use the NIF

Create a new Elixir module called ExampleNif:

defmodule ExampleNif do
  @on_load :load_nif

  def load_nif do
    :erlang.load_nif('./build/libexample_nif', 0)
  end

  def nif_add(_a, _b), do: :erlang.nif_error(:nif_not_loaded)

end

Let's review this. We:

Create an Elixir module called ExampleNif.
Specify that the load_nif/0 function should be called when the module is loaded.
Define a private function, load_nif/0, that loads the shared library containing the NIF.
Define a fallback function (nif_add/2) that will be called if the NIF fails to load.

Note: :erlang.load_nif loads and links the Zig compiled library to the Elixir module; and it also loads this function before the definition of our nif_add/2 fallback function. This is why we need to define the fallback function nif_add/2 with the same arity and name as the NIF function.

Because of Elixir's pattern-matching capabilities, the fallback function will handle calls to nif_add/2 if the NIF is not loaded, and if the NIF is loaded, the NIF function will handle the call as defined in load_nif/0.

Verifying That the NIF Works

To verify that the NIF is working as expected, create a simple test program in Elixir:

defmodule ExampleNifTest do
  use ExUnit.Case

  test "add/2 NIF" do
    assert ExampleNif.nif_add(1, 2) == 3
    assert ExampleNif.nif_add(-5, 10) == 5
  end
end

Run the test suite using mix test. If the tests pass, it indicates that the NIF is functioning correctly, and our addition is done through our Zig compiled library rather than on Elixir code directly.

Considerations

It is amazing that we can use code from another language through NIF. However, there are a few things you might want to be aware of, starting with the parameters we initially declared on our Zig library:

export fn nif_add(
    env: ?*erl.ErlNifEnv,
    argc: c_int,
    argv: [*c]const erl.ERL_NIF_TERM,
) erl.ERL_NIF_TERM {
    var a: i32 = undefined;
    var b: i32 = undefined;
...
    _ = erl.enif_get_int(env, argv[0], &a);
    _ = erl.enif_get_int(env, argv[1], &b);

We are parsing the first two arguments and casting them into an integer. What happens if we pass a float value to this function? Try it out:

iex(3)> ExampleNif.nif_add(-100.0,-11)
-1431655777

We get junk values, as there are no type checks or logic to handle float values in our Zig function. These can cause unexpected bugs and issues that developers need to be more vigilant to catch.

Wrapping Up

In this post, we set up Zig for Elixir NIF development, wrote a simple NIF in Zig, and integrated the NIF with an Elixir application.

By leveraging the strengths of both Elixir and Zig, developers can create efficient, maintainable, and performant applications.

That said, NIFs should be used with caution and deliberately. Ensure you add the necessary tests and failsafes.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Validate Data in a Phoenix Application for Elixir

Roy Tomeij — 2023-10-10T00:00:00+00:00

In this first part of a two-part series, we'll explore how to avoid bad data and validate data at the boundary of a Phoenix application.

We'll use a few techniques to ensure that bad data doesn't degrade our application.

In part two, we'll specifically focus on leveraging Ecto under the hood to cast data.

Let's dive in!

Say No to Bad Data in Elixir

Bad data must be dealt with immediately, or it will spread throughout your system and degrade other data. Given how difficult and time-consuming it is to fix data issues, preventing bad data from entering your system is well worth the effort.

Here is what the official Elixir documentation has to say on the matter:

[...] when you don't validate the values at the boundary, the internals of your library are never quite sure which kind of values they are working with.

This advice does not only apply to libraries, but to any Elixir code. Every time you receive multiple options or work with external data, you should validate the data at the boundary and convert it to structured data. For example, if you provide a GenServer that can be started with multiple options, you want to validate those options when the server starts and rely only on structured data throughout the process life cycle. Similarly, if a database or a socket gives you a map of strings, after you receive the data, you should validate it and potentially convert it to a struct or a map of atoms.

But what is a boundary, and where does it live? The answer, dear reader, is mostly up to you. In a sense, boundaries exist wherever your functions accept unknown or unsafe data, to later be processed (once some assumptions about the data are made).

An Example of a Boundary in a Phoenix App

A typical example of a boundary in a Phoenix app is the boundary between the web layer and the business logic: parameters come in as JSON values and are parsed into maps with string keys. But a client is free to send any sort of data within the JSON payload: there could be incorrect keys, invalid values, extra key values, and so on.

Since you don't want to deal with this unwieldy data in every location within your app, the recommendation is to process this untrusted data in a single location and convert it into a well-known shape with validated content.

This is typically done in a Phoenix Context, where Ecto is leveraged under the hood to cast data (we'll explore this in our next post). Once this conversion has taken place, your domain logic can trust the content of the request payload without performing the same verification again.

More generally speaking, boundaries will typically crop up anywhere in your software where you accept some data of unknown quality and process it internally (typically over several iterations). Examples include:

GenServers - where data is sent and processed into state changes.
Web requests - which make their way to domain logic and database tables.
Console input - which needs to be processed into arguments provided to a CLI application.

Additionally, boundaries can also show up when data has different meaning in different contexts (often called bounded contexts): a Customer might, for example, have a billing address, and a User might have a username, but both would refer to the same person in the world. In effect, boundaries are everywhere, which makes these techniques very handy when they're skillfully deployed.

Let's now check out a few techniques to prevent bad data from degrading our application.

Pattern Matching and Guards in Elixir

At the very local level, we can leverage pattern matching and guards to ensure we're always working with the data we expect.

This type of defensive code is beneficial anywhere you add it, be it within domain code (such as a Phoenix Context), the interface layer (in a Phoenix Controller, for example), or even deep within a free-standing function in a script.

Case Clauses

Case clauses can be very helpful in ensuring we explicitly list the data we agree to handle, particularly if there's no catch-all clause:

case Account.setup(...) do
    {:ok, %Account{suspended: true}} -> ...
    {:ok, %Account{initialized: true}} -> ...
    {:error, ...} -> ...
end

The above code clearly communicates its intent: the only expected outcomes of the setup function are an account (that is either initialized or suspended) or an error. Further, the case where an account is suspended "supersedes" an initialized account since the pattern match comes first.

Any other return value isn't expected and is therefore considered a bug: we explicitly don't handle those other cases, as we wouldn't know how to (if we did, they'd have their own case clause as shown above). In the face of unexpected data, it's safest to crash so that the OTP system can restart the process from a clean "known good" state rather than propagate dirty data throughout a system.

Pattern Matching in Function Heads

Pattern matching in function heads is a great way to not only ensure that data conforms to your expectations, but also to communicate your intent.

Contrast:

defp suspend(%Account{} = account)

With:

defp suspend(%{suspended: _} = account)

These are very different functions: in the first, it's clear that an Account is expected, so the account variable can be safely used in functions expecting an Account instance, whereas those assurances can't be made for the second version. In the second example, we're relying on the presence of the suspended attribute and a variable name to infer the context. That's playing with fire: there's nothing preventing someone from calling the function with a %User{} struct (assuming it also has a suspended attribute).

That said, while matching in function heads is helpful and convenient, make sure you're not muddying the waters for the sake of convenience. Let's use the following example to explore the subject, even though it's not directly related to validation:

def handle_call(:checkout, %{workers: [h | t], monitors: monitors}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [h | t]}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
    overflow: overflow, overflow_max: max, worker_sup: sup,
    spec: spec, monitors: monitors})
  when overflow < max do
    # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
  overflow: overflow, overflow_max: max, waiting: waiting}) do
# ...
end

A lot of matching is going on here, but how much is to differentiate the function heads, and how much is for convenience (i.e., binding for later use)?

By leaving only the matches that differentiate the heads and moving other bindings to the function bodies, we can improve readability:

def handle_call(:checkout, %{workers: [_|_]} = state) do
  %{monitors: monitors} = state
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [_]}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
    overflow: overflow, overflow_max: max} = state) when overflow < max do
  %{worker_sup: sup, spec: spec, monitors: monitors} = state
  # ...
end

def handle_call(:checkout, state) do
  %{workers: [], idle_overflow: [], overflow: overflow,
  overflow_max: max, waiting: waiting} = state
  # ...
end

After refactoring the above (non-validation related) example code, it's now more obvious that:

The first function matches when there are workers in the list.
The second function matches when there are no workers, but there are values in the idle_overflow list.
The third function matches when there are no workers or idle_overflow, but the overflow hasn't reached its max level yet.
The fourth function is for the remaining case.

You'll also note that we've gone somewhat overboard on matching here, such as matching on empty worker lists, even though the first function would match in that case. This approach is a bit of defensive programming, as the code will stay functional if:

The function clauses are reordered (e.g., during a refactoring), and their ordering importance is overlooked.
Additional matching is added to the first clause, changing its match semantics. Keeping each function clause responsible for declaring the context it expects makes the code more resilient to change.

Avoiding excessive matching in function heads is also the approach recommended by José Valim:

FWIW, I tend to use this rule: if the key is necessary when matching the pattern, keep it in the pattern, otherwise, move it to the body. So I end up with code like this:

def some_fun(%{field1: :value} = struct) do
  %{field2: value2, field3: value3} = struct
  ...
end

Guard Clauses

Guard clauses should be used whenever the definition of "what data is valid" can be tightened to make your code safer by design and to prevent processing unexpected/invalid data:

def annotate(%Account{} = account, annotation) when is_binary(annotation)

case account do
  %Account{payment_method: payment_method} when not is_nil(payment_method) -> ...
end

This same approach can naturally also be used for with and cond statements.

Custom Guards

Declaring custom guards effectively communicates intent and prevents bad data, significantly improving your code.

First, you must declare the guard in a separate module located outside of the module(s) where you want to use the guard.

defmodule Account.Guards do
  defguard is_suspended(account) when is_struct(account, Account) and account.suspended
end

Now our code is even more expressive:

import Account.Guards

case fetch_account(...) do
    %Account{} = account when is_suspended(account) -> ...
    ...
end

And that's it!

Next Up

In this post, we looked at how to validate data at the boundary of an Elixir application. We also used a few pattern matching and guard clause techniques to reject bad data.

We've already covered a lot of ground: let's give it a rest for now. In the next article, we'll explore a few more options to ensure the data we work with remains squeaky clean, using Ecto.

See you then!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How To Reduce Reductions in Elixir

Roy Tomeij — 2023-09-28T00:00:00+00:00

In this article, we'll show how you can use Elixir's profile.eprof mix task to evaluate and improve code performance in your Elixir application. You'll see how we used the profiling mix task to lower reductions in our instrument/3 function and custom instrumentation functionality, both key parts of our Elixir integration.

Our Key Components

Before we explain how we used Elixir's profile.eprof mix task to improve our instrument function's performance, let's take a moment to understand what profile.eprof does.

Elixir's `profile.eprof` Mix Task

Elixir's profile.eprof mix task evaluates a piece of code with the eprof profiling tool enabled. When the code runs, it outputs a time profile report, which shows the number of calls to each function and the amount of time spent in each function.

AppSignal's `instrument` Function

In its most basic form, the instrument/3 function wraps a piece of code to gain performance insights:

Appsignal.instrument("slow", fn ->
  :timer.sleep(1000)
end)

The example above creates a new trace (named "slow" when there's no current trace) or a child span in a trace that's already running.

With AppSignal, we can save and query our application's performance samples to quickly find similar samples or retrieve older samples when debugging issues.

Profiling our `instrument` Function

To look into the Appsignal.Instrumentation.instrument/3 function, we evaluated the following code snippet through the eprof task:

mix profile.eprof -e "(1..1_000) |> Enum.each(fn _ -> Appsignal.Instrumentation.instrument(\"name\", \"category\", fn -> :ok end) end)"

For convenience, this is a formatted version of the executed code, which calls the instrument/3 function a thousand times to eliminate any outliers and gauge the average performance of functions called by the instrument function:

(1..1_000)
|> Enum.each(fn _ ->
  Appsignal.Instrumentation.instrument(\"name\", \"category\", fn -> :ok end)
end)

Interpreting Profiling Data

Running the profiler produced the following report:

# This report sample has been redacted for brevity

Profile results of #PID<0.315.0>
#                                                        CALLS     %  TIME µS/CALL
Total                                                    11701 100.0 53586    0.46
:application.get_env/2                                    4000  0.90   484    0.12
:ets.lookup/2                                             6000  7.62  4083    0.68
Appsignal.Nif._create_root_span/1                         1000 13.82  7404    7.40
Appsignal.Nif._close_span/1                               1000 29.76 15947   15.95

The report shows all function calls executed as a result of calling instrument/3, with the rightmost column showing the time per call in microseconds.

We then interpreted the profiling data for our instrument function to understand how our function performed and find places in our code to reduce the number of functions called by our instrument function.

Looking for Reductions

Looking back at our report, we can see that the Appsignal.Nif._close_span/1 and Appsignal.Nif._create_root_span/1 functions took the most time, at roughly 16 and 7 microseconds per call, respectively.

Both functions were called 1,000 times, which made sense because that equated to the number of started and stopped spans. Although some performance gains might be had here, we skipped these function calls, as they were called a reasonable amount of times.

One place above these is :ets.lookup/2, which only took 0.68 microseconds per call, but was called 6,000 times, or $6n$ (where $n$ is the number of calls to the instrument/3 function).

  def instrument(name, category, fun) do
    instrument(fn span ->
      _ =
        span
        |> @span.set_name(name)
        |> @span.set_attribute("appsignal:category", category)

      call_with_optional_argument(fun, span)
    end)
  end

Investigating the instrument/3 function revealed that the Span.set_name/1 function called :ets.lookup/2 twice.

Reducing Calls

Although AppSignal uses ets internally to store the currently open spans, from checking the configuration, it turned out that the lookups in the set_name/1 function actually happened.

Internally, the integration checked to see if set_name/1 was active before calling Span.set_name/2, the function used to set the span name.

def set_name(%Span{reference: reference} = span, name)
    when is_reference(reference) and is_binary(name) do
  if Config.active?() do
    :ok = @nif.set_span_name(reference, name)
    span
  end
end

However, this call wasn't needed, as calling this function with a nil value instead of a span is a no-op. When the integration isn't active, nils are passed around instead of spans so this extra check could be safely removed.

After making these changes to our code and re-running our profiler, we saw a reduction in the number of calls by $2n$, leaving $4n$ calls to the :ets.lookup/2 function:

:ets.lookup/2                                             4000  5.06  2458    0.61
Appsignal.Nif._create_root_span/1                         1000 16.30  7922    7.92
Appsignal.Nif._close_span/1                               1000 32.91 16001   16.00

Another lookup was caused by a duplicate call to Application.get_env/1. Since the application environment is stored in ets as well, each get_env/1 call also does a lookup.

Previously, the environment was checked twice, in both the active?/0 and valid?/0 functions. If either of these is false, the integration is disabled. The implementation looked like this:

  def active? do
    :appsignal
    |> Application.get_env(:config, @default_config)
    |> do_active?
  end

  defp do_active?(%{active: true}), do: valid?()
  defp do_active?(_), do: false
  def valid? do
    do_valid?(Application.get_env(:appsignal, :config)[:push_api_key])
  end

  defp do_valid?(push_api_key) when is_binary(push_api_key) do
    !empty?(String.trim(push_api_key))
  end

  defp do_valid?(_push_api_key), do: false

Exposing valid?/1 (which takes a configuration and only tests if the push API key is present) saved a call to Application.get_env/2, which internally called another ets lookup:

  def active? do
    :appsignal
    |> Application.get_env(:config, @default_config)
    |> active?
  end

  defp active?(%{active: true} = config) do
    valid?(config)
  end

  defp active?(_config), do: false

  def valid? do
    :appsignal
    |> Application.get_env(:config)
    |> valid?
  end

  defp valid?(%{push_api_key: key}) when is_binary(key) do
    !(key
      |> String.trim()
      |> empty?())
  end

  defp valid?(_config), do: false

Having re-run the profiler, we saw the number of calls to :ets.lookup/2 was $3n$:

:erlang.whereis/1                                         4000  1.88   757    0.19
Appsignal.Nif._set_span_attribute_string/3                1000  1.90   766    0.77
Appsignal.Tracer.create_span/3                            1000  2.02   813    0.81
Process.whereis/1                                         4000  2.17   875    0.22
Appsignal.Tracer.running?/0                               4000  2.30   926    0.23
:ets.lookup/2                                             3001  3.91  1576    0.53
Appsignal.Nif._create_root_span/1                         1000 17.12  6903    6.90
Appsignal.Nif._close_span/1                               1000 33.97 13696   13.70

There was one call to try and find the parent, another call to see if the PID at the time was ignored, and a last call to check if the integration was active at that time.

It might be possible to remove the check that sees if the integration is active, which happens before starting every span. However, this particular update only consists of performance enhancements, so changes in functionality aren't included.

Removing Unnecessary Calls

A line above the :ets.lookup/2 function is Appsignal.Tracer.running?/0, a function that's called to ensure ets is running before executing lookups, inserts, and deletions.

Because ets produces an ArgumentError when it's not active, a check was called to ensure the registry was running every time the table was evaluated.

For example, this included a function to find the current span for a process named Tracer.lookup/1:

def lookup(pid) do
  if running?(), do: :ets.lookup(@table, pid)
end

This patch removed those checks and added error handling around each function that called ets, removing Appsignal.Tracer.running?/0 completely:

def lookup(pid) do
  try do
    :ets.lookup(@table, pid)
  rescue
    ArgumentError -> []
  end
end

The Final Profile

To summarise, we utilized profile.eprof to locate and reduce $7n$ unnecessary calls, including:

Calls to :ets.lookup/2 from $6n$ to $3n$
All $4n$ calls to Appsignal.Tracer.running

This resulted in a small but significant improvement in our instrument function's performance.

Sample AppSignal's Performance Monitoring

The ability to save and search performance samples is just one of AppSignal's many developer-driven features designed to help you get the most out of your application's metrics. Developers say they also love us thanks to our:

Intuitive interface that is easy to navigate.
Simple and predictable pricing.
Developer-to-developer support.

If you experience any issues when using AppSignal for Elixir, our support team is on hand to help! And remember, if you're new to AppSignal, we'll welcome you onboard with an exceptionally delicious shipment of stroopwafels 🍪 😋

An Introduction to Exceptions in Elixir

Roy Tomeij — 2023-09-26T00:00:00+00:00

Exceptions are a core aspect of programming, and a way to signal when something goes wrong with a program. An exception could result from a simple error, or your program might crash because of underlying constraints. Exceptions are not necessarily bad, though — they are fundamental to any working application.

Let’s see what our options are for handling exceptions in Elixir.

Raising Exceptions in Elixir

The Elixir (and Erlang) community is generally quite amenable to exceptions: “let it crash” is a common phrase. This is due, in part, to the excellent OTP primitives in Erlang/Elixir. The OTP primitives allow us to create supervisors that manage and restart processes (or a group of related processes) on failure.

Exceptions can occur when something unexpected happens in your Elixir application. For example, division by zero raises an ArithmeticError.

iex> 1 / 0
** (ArithmeticError) bad argument in arithmetic expression: 1 / 0
    :erlang./(1, 0)
    iex:1: (file)

Exceptions can also be raised manually:

iex> raise "BOOM"
** (RuntimeError) BOOM
    iex:1: (file)

By default, raise creates a RuntimeError. You can also raise other errors by using raise/2:

defmodule Math do
  def div(a, b) do
    if b == 0, do: raise ArgumentError, message: "cannot divide by zero"
    a / b
  end
end

iex> Math.div(1, 0)
** (ArgumentError) cannot divide by zero
    iex:4: Math.div/2
    iex:3: (file)

A function call that doesn’t match a defined function raises an ArgumentError by default:

defmodule Math do
  def div(a, b) when b != 0 do
    a / b
  end
end

iex> Math.div(1, 0)
** (FunctionClauseError) no function clause matching in Math.div/2

    The following arguments were given to Math.div/2:

        # 1
        1

        # 2
        0

Handling Elixir Exceptions

Elixir provides the try-rescue construct to handle exceptions:

try do
  raise "foo"
rescue
  e in RuntimeError -> IO.inspect(e)
end

This prints %RuntimeError{message: "foo"} in the console but doesn’t crash anything. Use this when you want to recover from exceptions in Elixir. It is also possible to skip binding the variable if it is unnecessary. For example:

try do
  raise "foo"
rescue
  RuntimeError -> IO.puts("something bad happened")
end

In both of the above examples, we only rescue RuntimeError. If there is another error, it will still raise an exception. To rescue all exceptions raised inside the try block, use the rescue without an error type, like this:

try do
  1 / 0
rescue
  e -> IO.inspect(e)
end

Running the above prints %ArithmeticError{message: "bad argument in arithmetic expression"}. If you want to perform different actions based on different exceptions, just add more clauses to the rescue branch.

try do
  1 / 0
rescue
  RuntimeError -> IO.puts("Runtime Error")
  ArithmeticError -> IO.puts("Arithmetic Error")
  _e -> IO.puts("Unknown error")
end

While rescuing all errors (without using a specific type) sounds tempting, it is a good practice to rely on specific exceptions because:

You can perform different recovery tasks for different exceptions.
It saves you from future breaking changes or programming errors going unnoticed.

For example, consider the below function:

defmodule Config do
  def get(file) do
    try do
      contents = File.read!(file)
      parse!(contents)
    rescue
      _e -> %{some: "default config"}
    end
  end
end

It reads and parses a config file, returning the result. When written, it uses a generic clause to rescue from missing file-related errors. Let’s say that a while later, the input file gets corrupted somehow (e.g., a simple missing } or an extra , in a JSON file), and now there are parsing errors. Our function will still work without raising any issues, and we will be left guessing why it doesn’t work even though there’s an existing file.

Another common practice in the Elixir community is to use ok/error tuples to signal errors instead of raising exceptions (for both internal and external libraries). So, instead of returning a simple result or raising an exception on an issue, a function in Elixir will return {:ok, result} on success and {:error, reason} on failure.

This means that most of the time, a case block will be preferable to try/rescue. For example, the above File.read! can be replaced by:

case File.read(file) do
  {:ok, contents} -> parse!(contents)
  {:error, reason} -> %{some: "default config"}
end

In practice, you should reach out for try/raise only in exceptional cases, never as a means of control flow. This is quite different from some other popular languages like Ruby or Java, where unexpected operations usually raise an error, then handle control flow for cases like non-existent files.

Re-raising Exceptions

Sometimes, we just want to know that there is an exception but not rescue from it — for example, to log an exception before allowing the process to crash or to wrap the exception into something more useful/understandable to the user.

This is where reraise can be helpful, as it preserves an exception's existing stack trace:

try do
  1 / 0
rescue
  e ->
    IO.puts(Exception.format(:error, e, __STACKTRACE__))
    reraise e, __STACKTRACE__
end

Here, we use the __STACKTRACE__ to retrieve the original trace of the exception, including its origin and the full call stack.

In a real-world application, you might use this to report a metric / send an exception to a third-party exception tracking service, or log something informative to a third-party logging service. For example, you might send telemetry data to AppSignal under these circumstances.

In addition, it is also common practice to have custom exceptions for cases where unexpected things happen in libraries. reraise/3 can be useful here:

defmodule DivisionByZeroError do
  defexception [:message]
end

defmodule Math do
  def div(a, b) do
    try do
      a / b
    rescue
      e in ArithmeticError ->
        reraise DivisionByZeroError, [message: e.message], __STACKTRACE__
    end
  end
end

Now using Math.div(1, 0) will raise a DivisionByZeroError instead of a generic ArithmeticError.

Rescue And Catch in Elixir

In Elixir, try/raise/rescue and try/throw/catch are different. raise and rescue are used for exception handling, whereas throw and catch are used for control flow.

A throw stops code execution (much like a return — the difference being that it bubbles up until a catch is encountered). It is rarely needed and is only an escape hatch to be used when an API doesn’t provide an option to do something.

For example, let's say that there’s an API that produces numbers and invokes a callback:

defmodule Producer do
  def produce(callback) do
    Enum.each((1..100) , fn x -> callback.(x) end)
  end
end

(This is just an example. Assume that with the real API, it is much more expensive to produce each number, and it produces an infinite list of numbers.)

We want to find the first number divisible by 13, since we don’t have any other apparent way of finding that number and stopping the production at that point. Let’s see how we can use throw/catch to do this:

try do
  Producer.produce(fn x ->
    if rem(x, 13) == 0 do
      throw(x)
    end
  end)
catch
  x -> IO.puts("First number divisible by 13 is #{x}")
end

It's worth stating that this is a bit of a contrived case, and most good APIs are designed in such a way that you never need to reach out for throw/catch.

After and Else Blocks in Elixir

You can use after and else blocks with all try blocks. An after block is called after processing all other blocks related to the try, regardless of whether any of the blocks raised an error. This is useful for performing any required clean-up operations.

For example, the below code creates a new producer and makes some results. If there’s an error during production, it will raise an exception. But the after block ensures that the producer is still disposed of regardless.

producer = Producer.new
try do
  Producer.produce!(producer)
after
  Producer.dispose(producer)
end

On the other hand, an else block is called only if the try block is completed without raising an error. The else block receives the try block's result. The return value from the else block is the final return value of the try. For example:

try do
  File.read!("/path/to/file")
rescue
  File.Error -> :not_found
else
  "a" -> :a
  _other -> :other
end

In the above code, if the file exists with content a, the result will be :a. The result is :not_found if the file doesn’t exist, and :other if the content is anything else.

Exceptions and Processes

No discussion about Elixir exceptions is complete without examining their impact on processes. Any unhandled exceptions cause a process to exit.

With this in mind, let’s revisit the “let it crash” strategy. If we don’t handle an exception from a process, it will crash. This is good in a way because:

Since all processes are separate from each other, an exception or unhandled crash from one process can never affect the state of another process.
In most sophisticated Elixir applications, all the processes run under a supervision tree. So, an unexpected exit will restart the process (depending on the supervision strategy) and any linked processes with a clean slate.

In most cases, intermittent issues will resolve themselves in the next run. This is much easier (and usually also cleaner) than handling each failure separately and performing a recovery step.

If you want to learn more about supervisors, I suggest the official Elixir Supervisor and Application guide as a great starting point.

Monitoring Exceptions

As we've seen, exceptions are a fundamental part of any application. Handling them is good, but sometimes, letting them crash a process is even better.

But in all cases, it is better to monitor exceptions happening in the real world so that you can take action if there’s something concerning (for example, a developer error that crashes an app).

This is where AppSignal for Elixir can help. Once set up, it automatically records all errors and can also trigger alerts based on predefined conditions.

Here's an example of an individual error you can get to from the "Errors" -> "Issue list" in AppSignal for debugging:

Check out the AppSignal for Elixir installation guide to get started.

Wrapping Up

In this post, we explored how errors are treated in Elixir and how to recover from them. We also saw that sometimes it is better to just let a process crash and be restarted through the supervisor than to manually perform recovery steps for all possible exceptions.

Elixir’s API makes a clear distinction between functions that raise an exception (usually ending in !) and functions that return a success/error tuple. If you are a library author, it is better to provide both options for users.

Reach out for the tuple-based methods when you need to handle error cases separately. In Elixir, we rarely use try blocks for control flow — the ! functions are for when we want a process to crash on unexpected events.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Phoenix 1.7 for Elixir: Edit a Form in a Modal

Roy Tomeij — 2023-09-12T00:00:00+00:00

In part one of this series, we introduced the CoreComponents that get generated when bootstrapping a new Phoenix project. In part two, we implemented a create modal.

Now, we will implement an edit modal.

You can continue following along with our companion repo.

Editing a Form in a Modal

You will first notice that each item will need a different changeset. We want to edit each item, so we need to be able to build a changeset from a different struct each time.

You could do this by iterating over all of the items in mount and rendering a different modal for every row, but this won't work at all. You would have to have one changeset per assign, which doesn't work when you have a list to add to. It would also mean a lot more HTML because you'd render the whole modal once per row. It's an all-round bad idea.

Instead, we need a way to build the correct changeset based on the item we click on. We can do that by using another JS function — push. This will push an event to the backend, along with any attributes that we want to send. If we add an edit button per row, the click action can push an event to the backend with the pet_id as a param. Then, we can select the pet from the list of assigns and build a changeset out of it.

First, add the button to the markup. It might be nice to use an icon for this, so let's take a quick detour to icons.

Icons in Phoenix

Phoenix 1.7 ships with a vendored heroicons library and an <.icon> component in CoreComponents. It works by supplying the name of an icon as a name attr, like so:

<.icon name="hero-cpu-chip" />

The names for the available icons are the filenames contained in the following path: assets/vendor/heroicons/optimized/20/solid/. Looking at the files doesn't tell us much about what they look like because we just see svg markup.

What would be cool is if we could render a dev-only route that displays all icons on one page. Then when we consider using an icon, we can go to that page and peruse them all at our leisure.

First, let's add the route:

# in lib/petacular_web/router.ex
live("/storybook", PetacularWeb.Pages.StoryBookLive, :show)

Then, we can make the necessary PetacularWeb.Pages.StoryBookLive module. We'll now write a function that generates all the icon names from the files in the assets folder, then iterate over them and create an icon from each one. This will give us a dynamic list of icons to render.

Here are the icon_names (this assumes you will start your server from the project's route):

# in lib/petacular_web/pages/story_book_live.ex

defp icon_names() do
  (File.cwd!() <> "/assets/vendor/heroicons/optimized/20/solid")
  |> Path.expand()
  |> File.ls!()
  |> Enum.map(fn path ->
    "hero-" <> String.replace(path, ".svg", "")
  end)
end

Then the markup:

# in lib/petacular_web/pages/story_book_live.ex
@impl true
def render(assigns) do
  ~H"""
  <div>
    <h1 class="text-xl mb-4 font-semibold">Storytime</h1>
    <div class="flex flex-col flex-wrap space-evenly">
      <%= for icon_name <- icon_names() do %>
        <section class="p-4 outline outline-1 mb-2 rounded">
          <h2 class="font-semibold"><%= icon_name %></h2>
          <div class="flex space-x-2 p-y-2 p-x-4 mr-2 my-2">
            <CoreComponents.icon name={icon_name} />
            <p>&ltCoreComponents.icon name="<%= icon_name %>"/&gt</p>
          </div>
          <div class="flex space-x-2 p-y-2 p-x-4 mr-2 my-2">
            <CoreComponents.icon name={icon_name<> "-mini"} />
            <p>&ltCoreComponents.icon name="<%= icon_name %>-mini"/&gt</p>
          </div>
          <div class="flex space-x-2 p-y-2 p-x-4 mr-2 my-2">
            <CoreComponents.icon name={icon_name<> "-solid"} />
            <p>&ltCoreComponents.icon name="<%= icon_name %>-solid"/&gt</p>
          </div>
        </section>
      <% end %>
    </div>
  </div>
  """
end

There is one more thing to do, though. Tailwind will purge all classes it doesn't see being used when the app is built. Usually, this is great because it means the bundle size is smaller, with more lightweight pages. However, here, it is going to bite us. When you refer to classes dynamically, Tailwind doesn't see those classes being used, so it purges them. Tailwind's docs warn about this.

We need to tell Tailwind not to purge all the icon modules so we can render them. We do that by adding a line of config into tailwind.config.js, like so:

safelist: [{ pattern: /hero\-.*/ }],

Now, we can head to http://localhost:4000/dev/storybook and see all the icons. See this commit for all of the changes.

Back to Editing Our Form

Okay, now we can select our edit icon and put it on the page.

<PetacularWeb.CoreComponents.icon name="hero-pencil-square-solid" class="mr-2" />

We will put this in a button and add a phx-click that opens our edit modal for us. All this can live in the homepage we used in parts one and two.

# in lib/petacular_web/pages/home_live.ex

  @impl true
  def render(assigns) do
    ~H"""
    ...

    <div class="w-50 mb-4">
      <%= for pet <- @pets do %>
        <div class="flex">
          <button phx-click={open_edit_modal(pet.id)}>
            <PetacularWeb.CoreComponents.icon name="hero-pencil-square-solid" class="mr-2" />
          </button>
          <p>Name: <span class="font-semibold"><%= pet.name %></span></p>
        </div>
      <% end %>
    </div>

    ...
    """
  end

We also need to create our Edit modal. This will be similar to our Create modal, but a bit different, so we'll just create a new modal.

We can put this in our ~H component just before the other modal:

# in lib/petacular_web/pages/home_live.ex

  @impl true
  def render(assigns) do
    ~H"""
  <PetacularWeb.CoreComponents.modal id="edit_modal">
    <h2>Edit a pet.</h2>

    <PetacularWeb.CoreComponents.simple_form for={@edit_changeset} phx-submit="edit_pet">
      <PetacularWeb.CoreComponents.input
        label="Name"
        id="edit_name"
        field={@edit_changeset[:name]}
        value={@edit_changeset[:name].value}
      />
      <:actions>
        <PetacularWeb.CoreComponents.button>
          Save
        </PetacularWeb.CoreComponents.button>
      </:actions>
    </PetacularWeb.CoreComponents.simple_form>
  </PetacularWeb.CoreComponents.modal>
    ...
    """
  end

Now we need to add a changeset to the assigns. Initially, we can put any changeset in mount because when we open the modal, we are going to seed it:

# in lib/petacular_web/pages/home_live.ex

@impl true
def mount(_params, _session, socket) do
  default_assigns = %{
    pets: Repo.all(Petacular.Pet),
    edit_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{})),
    create_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
  }

  {:ok, assign(socket, default_assigns)}
end

Add the `open_edit_modal` Function

Now let's implement the open_edit_modal function. This has to do two things:

Open the modal.
Trigger a message to the backend so we can seed the changeset.

# in lib/petacular_web/pages/home_live.ex

defp open_edit_modal(pet_id) do
  %JS{}
  |> JS.push("open_edit_modal", value: %{pet_id: pet_id})
  |> PetacularWeb.CoreComponents.show_modal("edit_modal")
end

The handler for this event needs to select the relevant pet from the list of pets and put that into the changeset:

# in lib/petacular_web/pages/home_live.ex

@impl true
def handle_event("open_edit_modal", %{"pet_id" => id}, socket) do
  pet = Enum.find(socket.assigns.pets, &(&1.id == id))

  new_assigns = %{
    edit_form: Phoenix.Component.to_form(Petacular.Pet.edit_changeset(%{}, pet))
  }

  {:noreply, assign(socket, new_assigns)}
end

When we open the modal, the page will re-render the form because the changeset has changed, and the form will be seeded with the correct data. We can verify this by using |> IO.inspect(limit: :infinity, label: "") on the form value:

value={@edit_changeset[:name].value |> IO.inspect(limit: :infinity, label: "edit for name:")}

Debugging an Issue with the `open_edit_modal`

If you open the modal, you will see the correct value printed. But there is a problem — it's not showing on the page! What on earth could be the issue? This one is a doozy, so I will save you some hours of debugging.

The function that we use to open our modal has this line, which focuses the first "focussable" element in the modal:

|> JS.focus_first(to: "##{id}-content")

This is done for accessibility, and so is generally a good idea.

The input happens to be the first focussable thing in our edit form. Phoenix also ensures that the client is the source of truth for an input's value:

For any given input with focus, LiveView will never overwrite the input's current value, even if it deviates from the server's rendered updates.

So what happens in our case? We push an asynchronous message to the backend, which changes an assign (the edit form), causing a re-render — |> JS.push("open_edit_modal", value: %{pet_id: pet_id}). Then we open the modal with JavaScript, but because the message to the server is async, the modal opens before we get a reply. The first focussable element is the input field, so that gets focus, then the server responds. This would normally re-render the input field, but now won't, because the input has focus!

Fixing the Issue

We've done everything right, yet are left adrift. What are our options?

Remove the auto-focus capabilities of the modal.
Have the edit modal focus on something that is not the input first.
Somehow make the form opening synchronous to the server message.

I honestly don't know which is better, but let's reason them out. One is easy to do — just remove this line from the show_modal function — but may have accessibility implications, which makes it a non-starter.

The second option seems reasonable at first. We could maybe set the tabindex on the heading, but mdn recommends the following:

If an element can be focused using the keyboard, then it should be interactive; that is, the user should be able to do something to it and produce a change of some kind (for example, activating a link or changing an option).

So that's out.

The third option is possible, but requires some ceremony. Instead of opening the modal with JS, you could have the backend trigger a JS event that opens the modal when it's finished seeding the changeset. That requires adding a handler in JS to listen for the event and also means that the modal open is slower, because it requires at least one round trip to the server. For me, that is out as well.

The solution? A secret fourth option — set the value of the field with JS. This means the field will open quickly, be set to the correct value, and auto-focus.

To support that, we alter our open_edit_modal function to accept the name, then use JS.set_attribute to set the field value.

...
<button phx-click={open_edit_modal(pet.id, pet.name)}>
  <PetacularWeb.CoreComponents.icon name="hero-pencil-square-solid" class="mr-2" />
</button>
...

defp open_edit_modal(pet_id, pet_name) do
  %JS{}
  |> JS.push("open_edit_modal", value: %{pet_id: pet_id})
  |> JS.set_attribute({"value", pet_name}, to: "#edit_name")
  |> PetacularWeb.CoreComponents.show_modal("edit_modal")
end

Implementing the Update in Our Phoenix Application

Now the only thing left to do is implement the edit_pet handler. This is similar to the create version, where we flash an error and close the modal on success. We first want to select the pet we are editing, which means we need the pet's id. How can we get that?

The easiest way is to use a hidden input on the form. That way, when the form is submitted, the pet id will also be sent. To do that, we need to add the hidden input:

<%= Phoenix.HTML.Form.hidden_input(f, :id, id: "edit_pet_id_input") %>

And set the value when we open the modal:

defp open_edit_modal(pet_id, pet_name) do
  %JS{}
  |> JS.push("open_edit_modal", value: %{pet_id: pet_id})
  |> JS.set_attribute({"value", pet_name}, to: "#edit_name")
  |> JS.set_attribute({"value", pet_id}, to: "#edit_pet_id_input")
  # ^^^ this line ^^^^
  |> PetacularWeb.CoreComponents.show_modal("edit_modal")
end

We will see the id of the pet appear in the params, allowing us to select the pet we are editing from the assigns:

@impl true
def handle_event("edit_pet", %{"pet" => %{"id" => id} = params}, socket) do
  pet = Enum.find(socket.assigns.pets, &(&1.id == String.to_integer(id)))

  case Repo.insert(Petacular.Pet.edit_changeset(params, pet)) do
    {:error, message} ->
      {:noreply, socket |> put_flash(:error, inspect(message))}

    {:ok, _} ->
      new_assigns = %{
        pets: Repo.all(Petacular.Pet),
        edit_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
      }

      socket =
        socket
        |> assign(new_assigns)
        |> push_event("close_modal", %{to: "#close_modal_btn_edit_modal"})

      {:noreply, socket}
  end
end

See this commit for all the relevant changes.

And with that, we are done!

Wrapping Up

This concludes our three-part series in which we took a fresh Phoenix 1.7 application and built a create and edit modal for it.

Hopefully, this gives you some new ideas you can extend and implement for your own apps.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Writing a Custom Credo Check in Elixir

Roy Tomeij — 2023-08-29T00:00:00+00:00

Static code analysis is an important tool to ensure a project meets the right code standards and quality. In Elixir, the most popular package for this is Credo. Not only does it offer dozens of pre-made checks, but it also allows you to create your own.

In this article, we will walk you through creating a Credo check. We will see how to write the code, enable the check in the Credo config, and make it nice to use.

Let’s start!

Why Create Credo Checks?

There are many things you can create Credo checks for. Some examples are:

To check a timestamp in migration files and see if it shows the correct date
To forbid calling business logic from migration files
To disallow referencing or aliasing MyAppWeb modules from MyApp
To ensure some naming conventions, for example, forbidding the is_ prefix for functions (prefer active? to is_active)

Now let's dive into a real-world example with an Elixir app.

Getting Started with Credo for Elixir

Let's first add Credo to an Elixir project. We will start with a new empty mix project:

$ mix new my_app

Then we will cd into the newly created my_app and add Credo to the dependencies in mix.exs, so it looks like this:

defp deps do
  [{:credo, "~> 1.7", only: [:dev, :test], runtime: false}]
end

After that, we can run mix deps.get, then run Credo with mix credo. The output is not very interesting — our project is empty, so it does not violate anything. Let's quickly change that. Open lib/my_app.ex and change it to this:

defmodule MyApp do
  @moduledoc """
  Documentation for `MyApp`.
  """

  def test do
    x = 1; y = 2
  end
end

Now when we run mix credo, we get a nice error:

Checking 3 source files ...

  Code Readability
┃
┃ [R] ↗ Don't use ; to separate statements and expressions
┃       lib/my_app.ex:7:10 #(MyApp.test)

Please report incorrect results: https://github.com/rrrene/credo/issues

Analysis took 0.1 seconds (0.07s to load, 0.1s running 55 checks on 3 files)
3 mods/funs, found 1 code readability issue.

If we are unsure what that means, we can ask for details:

$ mix credo explain lib/my_app.ex:7

  MyApp
┃
┃   [R] Category: readability
┃    ↗  Priority: high
┃
┃       Don't use ; to separate statements and expressions
┃       lib/my_app.ex:7:10 (MyApp.test)
┃
┃    __ CODE IN QUESTION
┃
┃     5
┃     6   def test do
┃     7     x = 1; y = 2
┃                ^
┃     8   end
┃     9 end
┃
┃    __ WHY IT MATTERS
┃
┃       Don't use ; to separate statements and expressions.
┃       Statements and expressions should be separated by lines.

The output above is clipped. Run it yourself to read the whole explanation and to see it in color too!

Anatomy of a Credo Check

Credo checks are divided into categories. The one above falls under "readability". We also have "consistency", "refactor", "design", and "warning". Each check also has its own priority (set to 'high' in the example above), name, and explanation.

The check is an Elixir module. Built-in checks are kept in the Credo repository.

The check in question is defined in this file.

It starts with use Credo.Check, followed by quite a lot of passed configs. An identifier of a check is defined, its priority, tags, and a long string with an explanation — we saw this when we ran mix credo explain.

This is followed by a definition of the run function, taking a source file and some params as input, as the entry point for the check. It will run for every file where the check is enabled, passing the file as the first argument.

Other than that, the check defines a bunch of private functions. Three versions of collect_issues are responsible for finding lines with a semicolon token. And if found, the issues accumulator is updated with a relevant line and column number, as well as an appropriate message and trigger.

An important thing to note is Credo.Code.to_tokens(source_file) — it splits the source file contents into tokens and feeds those tokens to the collect_issues functions.

If we put IO.inspect there, we see that our file from the above listing looks like this (parsed as tokens):

[
  {:identifier, {1, 1, 'defmodule'}, :defmodule},
  {:alias, {1, 11, 'MyApp'}, :MyApp},
  {:do, {1, 17, nil}},
  {:eol, {1, 19, 1}},
  {:at_op, {2, 3, nil}, :@},
  {:identifier, {2, 4, 'moduledoc'}, :moduledoc},
  {:bin_heredoc, {2, 14, nil}, 2, ["Documentation for `MyApp`.\n"]},
  {:eol, {4, 6, 2}},
  {:identifier, {6, 3, 'def'}, :def},
  {:do_identifier, {6, 7, 'test'}, :test},
  {:do, {6, 12, nil}},
  {:eol, {6, 14, 1}},
  {:identifier, {7, 5, 'x'}, :x},
  {:match_op, {7, 7, nil}, :=},
  {:int, {7, 9, 1}, '1'},
  {:";", {7, 10, 0}},
  {:identifier, {7, 12, 'y'}, :y},
  {:match_op, {7, 14, nil}, :=},
  {:int, {7, 16, 2}, '2'},
  {:eol, {7, 17, 1}},
  {:end, {8, 3, nil}},
  {:eol, {8, 7, 1}},
  {:end, {9, 1, nil}},
  {:eol, {9, 4, 1}}
]

Indeed, we have a {:";", {7, 10, 0}} token, which is easy to match. It means that there is a semicolon in line 7, column 10 — which is exactly where the semicolon is in our code.

Armed with that knowledge, we can start writing our own check.

Our Very Own Credo Check in Elixir

We will write a check forbidding us from doing "bare imports". What I mean by that is calls like import Ecto.Changeset.

Why should we do this, though? Well, when you import the entire Ecto.Changeset module, and then use specific functions from that module, it can be hard for future collaborators (or future you) to determine where those functions were first defined.

However, in Elixir, you can specify a list of particular functions to import. So this would be fine:

import Ecto.Changeset, only: [cast: 4]

To start, we will add a "bare" import statement to our MyApp code. Remember to also add Ecto to the dependencies, otherwise the code will not compile.

defmodule MyApp do
  @moduledoc false
  import Ecto.Changeset
end

Now we need the actual check too. Credo offers a mix task to generate a new check. It adds example content, which might sometimes be useful. For our purpose, it would be confusing to start with the example code and then remove most of it, so we will build our check from scratch.

If you want to use the generator in the future, here's how:

$ mix credo.gen.check lib/credo/precise_imports.ex

Instead, we will create the file skeleton ourselves. In the same location, let's start with this:

defmodule Credo.Check.Readability.PreciseImports do
  @moduledoc false
  use Credo.Check,
    base_priority: :medium,
    explanations: [check: "Use :only with all imports"]

  @impl true
  def run(source_file, params) do
    []
  end
end

Write Some Tests in Credo

To check if it works, we need to write some tests. Luckily for us, Credo makes this really easy.

defmodule Credo.Check.Readability.PreciseImportsTest do
  use Credo.Test.Case

  @described_check Credo.Check.Readability.PreciseImports

  test "it should not raise issues" do
    """
    defmodule TestModule do
      import MyApp.Helpers, only: [hello: 0]
    end
    """
    |> to_source_file()
    |> run_check(@described_check)
    |> refute_issues()
  end

  test "it should report a violation" do
    """
    defmodule TestModule do
      import MyApp.Helpers
    end
    """
    |> to_source_file()
    |> run_check(@described_check)
    |> assert_issue()
  end
end

We also need to start the Credo application in our test_helper.ex:

Credo.Application.start([], [])

Now we have one test passing and one failing because we have not implemented the check yet. How should we do that?

If we try to use to_tokens, it might be quite hard to catch the import without the only option. Tokens are always a flat list of, well, tokens. It's hard to pattern match to some cases, based on the option passed to the import call.

[
  {:identifier, {1, 1, 'defmodule'}, :defmodule},
  {:alias, {1, 11, 'TestModule'}, :TestModule},
  {:do, {1, 22, nil}},
  {:eol, {1, 24, 1}},
  {:identifier, {2, 3, 'import'}, :import},
  {:alias, {2, 10, 'Ecto'}, :MyApp},
  {:., {2, 15, nil}},
  {:alias, {2, 16, 'Changeset'}, :Helpers},
  {:eol, {2, 23, 1}},
  {:end, {3, 1, nil}},
  {:eol, {3, 4, 1}}
]

ASTs in Elixir

We need a different approach. Fortunately, Elixir provides a "smarter" representation of the code as an AST (abstract syntax tree). We can get it by using Credo.Code.ast. Here is how the results look for our simple module:

> source = """
defmodule TestModule do
  import Ecto.Changeset
end
"""
> Credo.Code.ast(source)
{:ok,
 {:defmodule, [line: 1, column: 1],
  [
    {:__aliases__, [line: 1, column: 11], [:TestModule]},
    [
      do: {:import, [line: 2, column: 3],
       [{:__aliases__, [line: 2, column: 10], [:Ecto, :Changeset]}]}
    ]
  ]}}

And when we specify import with only: [cast: 4]:

> source = """
defmodule TestModule do
  import Ecto.Changeset, only: [cast: 4]
end
"""
> Credo.Code.ast(source)
{:ok,
 {:defmodule, [line: 1, column: 1],
  [
    {:__aliases__, [line: 1, column: 11], [:TestModule]},
    [
      do: {:import, [line: 2, column: 3],
       [
         {:__aliases__, [line: 2, column: 10], [:Ecto, :Changeset]},
         [only: [cast: 4]]
       ]}
    ]
  ]}}

This might look quite difficult to understand. Fortunately, Credo offers a prewalk function to make our job easier.

The `prewalk` Function

Let's change our check to use the prewalk function:

defmodule Credo.Check.Readability.PreciseImports do
  @moduledoc false
  use Credo.Check,
    base_priority: :normal,
    explanations: [check: "Use :only with all imports"]

  @impl true
  def run(source_file, params) do
    source_file
    |> Credo.Code.prewalk(&traverse(&1, &2, IssueMeta.for(source_file, params)))
  end

  defp traverse(ast, issues, issue_meta), do: {ast, add_issue(issues, issue(ast, issue_meta))}

  defp add_issue(issues, nil), do: issues
  defp add_issue(issues, issue), do: [issue | issues]

  defp issue({:import, meta, [{:__aliases__, _, _}]}, issue_meta) do
    issue_for(issue_meta, meta[:line])
  end

  defp issue({:import, meta, [{:__aliases__, _, _}, opts]}, issue_meta) do
    if Keyword.has_key?(opts, :only), do: nil, else: issue_for(issue_meta, meta[:line])
  end

  defp issue(_, _), do: nil

  defp issue_for(issue_meta, line_no) do
    format_issue(
      issue_meta,
      message: "Use :only with import statements",
      line_no: line_no
    )
  end
end

Let's break this down.

def run(source_file, params) do
  source_file
  |> Credo.Code.prewalk(&traverse(&1, &2, IssueMeta.for(source_file, params)))
end

We take the source_file here and feed it to the Credo.Code.prewalk function. This accepts a function that is called for every node of the AST we traverse. If we were to log what arguments are passed by the prewalk function, the second one is an accumulator where we should store a list of offences detected by our check (starting with an empty list).

As for the first argument, it's the current node of the AST. In the first run, it will be the whole tree, and in the subsequent calls, it will pass the inner leaves.

The first run:

{:defmodule, [line: 1, column: 1],
 [
   {:__aliases__, [line: 1, column: 11], [:TestModule]},
   [
     do: {:import, [line: 2, column: 3],
      [
        {:__aliases__, [line: 2, column: 10], [:MyApp, :Helpers]},
        [only: [hello: 0]]
      ]}
   ]
 ]}

Second:

{:__aliases__, [line: 1, column: 11], [:TestModule]}

In the third step, it's just the name of the module:

:TestModule

There's nothing here to go deeper into, so we go to the next "sibling" node:

[
  do: {:import, [line: 2, column: 3],
   [
     {:__aliases__, [line: 2, column: 10], [:MyApp, :Helpers]},
     [only: [hello: 0]]
   ]}
]

It will go down deeper, but at some point before it finishes, the argument will be very useful for us. It will look like this:

{:import, [line: 2, column: 3],
 [{:__aliases__, [line: 2, column: 10], [:MyApp, :Helpers]}, [only: [hello: 0]]]}

Before we move to the traverse function, there is also an IssueMeta.for(source_file, params) call. In fact, this is Credo.IssueMeta.for/2, but use Credo.Check creates an alias for it. The function returns a metadata structure, which will help Credo identify in which file the issue is found. We don't need to know more about it, just use it: pass it to traverse and then to issue.

defp traverse(ast, issues, issue_meta), do: {ast, add_issue(issues, issue(ast, issue_meta))}

The `traverse` Function

As we know from before, traverse accepts an AST and accumulator of found issues (initially empty). The third argument is the issue_meta required by Credo.

Implementing this function is simple: it returns the same AST node (so prewalk can take it and traverse it further) and changes the accumulator (or not) using the add_issue function.

defp add_issue(issues, nil), do: issues
defp add_issue(issues, issue), do: [issue | issues]

So, if the "current issue" is a nil, just return the list of issues. But if it's something else, prepend it to the list.

The `issue` Function

Now we need to look at the issue function, which is finally responsible for detecting the actual issue. This function takes the AST node as the first argument and Credo's issue_meta as the second. If we detect an issue, it returns the relevant issue information (issue_for, or nil otherwise).

Remember how the AST of an import with the only option looks? We will match against it now:

{:import, [line: 2, column: 3],
 [{:__aliases__, [line: 2, column: 10], [:MyApp, :Helpers]}, [only: [hello: 0]]]}

The first "wrong" variant is when no options are passed to the import statement (import Ecto.Changeset) at all. We will match with this and return a non-nil result:

defp issue({:import, meta, [{:__aliases__, _, _}]}, issue_meta) do
  issue_for(issue_meta, meta[:line])
end

Another possible violation is when options are given, but they don't contain only, for example: import Ecto.Changeset, except: [from: 3]. We match the opts in the function definition, and then check if it contains the required key. If not, we add an issue.

defp issue({:import, meta, [{:__aliases__, _, _}, opts]}, issue_meta) do
  if Keyword.has_key?(opts, :only), do: nil, else: issue_for(issue_meta, meta[:line])
end

We also need one definition for all other cases. We pass the AST nodes to the function, including ones not related to the import. These nodes cannot be the cause of the issue we are looking for, so it's a simple "catch-all" variant:

defp issue(_, _), do: nil

The last thing in the module is the issue_for functions. These take the meta generated by IssueMeta.for and pass it to Credo's format_issue function, along with the message and the line number.

defp issue_for(issue_meta, line_no) do
  format_issue(
    issue_meta,
    message: "Use :only with import statements",
    line_no: line_no
  )
end

We can run our tests now to verify that they pass: the check detects issues correctly.

Enabling the Credo Check in Elixir

As we know that our check works fine (because our tests pass), we can now run mix credo on our project. However, even though we know the code is using a "bare import", Credo is still not reporting the issue! To fix that, we need to add our custom check to the Credo config first. In this project, we don't have the config file yet: we have to create it with mix credo gen.config.

This will create quite a big .credo.exs file with the default configuration. We can skip most of it. Just find a requires key and add the path to our check there:

requires: ["lib/credo/precise_imports.ex"],

Then we need to add the check to the enabled section:

checks: %{
  enabled: [
    {Credo.Check.Readability.PreciseImports, []},
    [...]

With that in place, we can now run mix credo again.

$ mix credo
Checking 1 source file ...

  Code Readability
┃
┃ [R] → Use :only with import statements
┃       lib/my_app.ex:3 #(MyApp)

If we change it to use only, the check will pass. If we change it to use except, but not only, the check will fail again (we should probably add a test for this).

Improving Our Explanation for Credo

One last thing we should do is to improve the explanation we passed as an option to use Credo.Check. Change it to something more descriptive:

defmodule Credo.Check.Readability.PreciseImports do
  @moduledoc false
  use Credo.Check,
    base_priority: :normal,
    explanations: [
      check: """
      Using bare import statements, without specifying what we are
      importing makes it hard to reason about from where the function
      comes from...
      """
    ]

Now it will look better when we run mix credo explain:

$ mix credo explain lib/my_app.ex:3

  MyApp
┃
┃   [R] Category: readability
┃    →  Priority: normal
┃
┃       Use :only with import statements
┃       lib/my_app.ex:3 (MyApp)
┃
┃    __ CODE IN QUESTION
┃
┃     1 defmodule MyApp do
┃     2   @moduledoc false
┃     3   import MyApp.Helpers, except: [hello: 0]
┃     4 end
┃     5
┃
┃    __ WHY IT MATTERS
┃
┃       Using bare import statements, without specifying what we are
┃       importing makes it hard to reason about from where the function
┃       comes from...

And that's it!

Wrapping Up

In this post, we learned how to create a Credo check to enforce a custom code rule. We discussed using tokenization and AST generation to match bad code. Finally, we ran some tests and enabled the Credo check in Elixir.

Happy static analyzing!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Deep Dive into Subscriptions with Absinthe

Roy Tomeij — 2023-08-08T00:00:00+00:00

In this series, we've seen how to create GraphQL APIs in Elixir using Absinthe. So far, we have only discussed a one-way communication channel where the client makes the queries or mutations, and the server responds.

GraphQL also supports a long-running subscription between the client and the server where the server can notify the client of events. This can be very useful in multi-user scenarios where many users might interact with the same resource at the same time. For example, in a blogging application, the client can subscribe to the server to receive new comments as they are posted instead of having to poll the server for them.

This is now a two-way communication and the regular HTTP transport is not well suited for it. Let’s see how subscriptions work and how to create them.

Creating Subscriptions in Absinthe for Elixir

Creating subscriptions is very similar to creating query or mutation fields. We just need to create a field inside the subscription do ... end block of our schema.

Let’s create a subscription for observing new comments inside the schema:

# lib/my_app_web/schema.ex

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  # ...

  subscription do
    field :comment_added, :comment do
      arg :post_id, non_null(:id)

      config fn %{post_id: id}, %{context: context} ->
        {:ok, topic: "post:#{id}"}
      end
    end
  end
end

Let’s break this down. We add a field named comment_added inside the subscription block. The return type of this field is comment (defined using the object macro in Absinthe — see An Introduction to Absinthe for details). This field accepts an argument named post_id which is required. So far, everything is similar to what we would do when creating a field in a query.

The config macro is what is new here and it is used to configure the subscription field. This is also where we can perform other checks like authorization to accept/reject the subscription. The function passed to config receives the arguments passed to the field and the Absinthe.Resolution struct that contains information like context. The return value can be an {:ok, topic: topic} tuple for success or {:error, reason} tuple. The topic is what Absinthe uses to identify subscribers who should be notified of an event. We will get to it in a few minutes. But just for completeness, the topic can be a single string or a list containing multiple topics to subscribe to.

Publishing Data to Subscriptions in GraphQL

We have a subscription added to our schema. But even after subscribing, we won’t get back any data because we haven’t triggered any updates yet.

Trigger from Mutations

The easiest way to trigger the subscription is to use the trigger/2 macro inside the subscription field. It accepts the name of a mutation to trigger the subscription and a topic function. Assuming that we have a mutation named create_comment in our schema to create a comment, let’s update our subscription to trigger every time a comment is created.

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

    # ...

  subscription do
    field :comment_added, :comment do
            # arg and config ...

      trigger :create_comment, topic: fn comment ->
        "post:#{comment.post_id}"
      end
    end
  end
end

The function we pass as topic to trigger will be executed with the create_comment mutation result. It should return the name of the topic that this mutation will trigger.

Let’s see this in action. First, user A runs a subscription query like this:

subscription {
  commentAdded(postId: 1) {
    id
    body
    author {
      id
      name
    }
  }
}

Absinthe records the subscription, but the user doesn’t receive any data straight away.

Now, user B creates a new comment:

mutation {
  createComment(comment: { postId: 1, body: "nice post!" }) {
    id
  }
}

User B will receive the mutation response as usual based on the selections:

{
  "data": {
    "createComment": {
      "id": 1
    }
  }
}

Additionally, user A will also receive a response on their subscription. Since they selected many more fields when subscribing, the response will include them all regardless of the selections in the mutation.

{
  "data": {
    "commentAdded": {
      "id": 1,
      "body": "nice post!",
      "author": {
        "id": 1,
        "name": "User B"
      }
    }
  }
}

Note that with the current setup, user B won't actually receive any updates yet since the subscription wasn't made using a WebSocket transport. We will see how to set that up later in this post.

Translate Mutation Result to Subscription Result

If your mutation returns a different data structure from what the subscription is supposed to return, it is also possible to use a resolve inside the subscription to convert the result first.

For example, assume that instead of returning a comment, the create_comment mutation returns a complex object that also includes validation errors in case of failure. In that case, we can provide a resolver for the comment_added subscription field to fetch the comment before returning the result. We will also need to modify our trigger to create the topic from the response struct.

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  object :comment_mutation_result do
    field :comment, :comment
    field :errors, list_of(:string)
  end

  mutation do
    field :create_comment, :comment_mutation_result do
      arg :comment, non_null(:comment_create_input)

      resolve fn %{comment: attrs}, _resolution ->
        case MyAppWeb.Blog.create_comment(attrs) do
          {:ok, comment} ->
            {:ok, %{comment: comment, errors: []}}
          {:error, changeset} ->
            {:ok, %{comment: nil, errors: translate_changeset_errors(changeset)}}
        end
      end
    end
  end

  subscription do
    field :comment_added, :comment do
      arg :post_id, non_null(:id)

      config fn %{post_id: id} = args, %{context: context} ->
        {:ok, topic: "post:#{id}"}
      end

      trigger :create_comment, topic: fn
        %{comment: nil} -> []
        %{comment: comment} -> "post:#{comment.post_id}"
      end

      resolve fn %{comment: comment}, _args, _resolution ->
        {:ok, comment}
      end
    end
  end
end

Trigger from Application Code

The trigger macro works great as long as the only way to create a comment is through the GraphQL API. But that might not always be the case. For example, you might have a separate REST API or traditional Phoenix-based controllers/Live Views that create those comments. In that case, it is usually better to drop the trigger from the GraphQL API and publish notifications at a lower level from the application code.

This can be achieved using the Absinthe.Subscription.publish/3 method. A good place to put this might be inside the Phoenix Context method that creates the comment. Let’s do that now:

defmodule MyApp.Blog do
  # ...

  def create_comment(attrs) do
    %Comment{}
    |> Comment.changeset(attrs)
    |> Repo.insert()
    |> tap(fn
      {:ok, comment} ->
        Absinthe.Subscription.publish(MyAppWeb.Endpoint, comment, comment_created: "post:#{comment.post_id}")
      _ ->
        nil
    end)
  end
end

The name of the subscription field to trigger (comment_updated) is passed as a key inside the last argument to this function. The value of that key is the topic to target.

Note that you use both the trigger macro and publish/3. Just make sure that the code that calls publish is not being executed from the mutation referenced in the trigger. Otherwise, the user will receive two messages for a single comment — one from the trigger macro and another from publish.

Server-Side Setup with Phoenix PubSub

At the beginning of the post, we discussed that a simple one-way HTTP transport is not well-suited for subscriptions. Let's come back to that topic and see how we can set up the application to support a long-running two-way connection between the application and the client.

The easiest way to do it is using Phoenix PubSub which is present by default in all Phoenix applications. Let’s install Absinthe’s Phoenix helpers by adding the following dependency in mix.exs:

{:absinthe_phoenix, "~> 2.0"}

Next, configure the PubSub and add it to the supervision tree (this might already be configured for you if you generated the application using phx.new):

# config/config.exs
config :my_app, MyAppWeb.Endpoint, pubsub_server: MyApp.PubSub

# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      # ...
      # Start the PubSub system
      {Phoenix.PubSub, name: MyApp.PubSub},
      # Absinthe Subscription
      {Absinthe.Subscription, MyAppWeb.Endpoint},
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Note that we also added Absinthe.Subscription inside the supervision tree which keeps track of all the active subscriptions and takes care of delivering notifications to the subscribed clients. We pass the name of the endpoint as the only argument, but other configuration options are available — see Absinthe.Subscription.child_spec/1 for full details.

Next, use Absinthe.Phoenix.Endpoint inside your web application’s endpoint module:

# lib/my_app_web/endpoint.ex
defmodule MyAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :my_app
  use Absinthe.Phoenix.Endpoint

  # ...
end

It adds some utility methods to the Endpoint module used by Absinthe.Subscription to publish the results of the subscription.

Finally, use Absinthe.Phoenix.Socket inside the socket module.

# lib/spendra_web/channels/user_socket.ex
defmodule MyAppWeb.UserSocket do
  use Phoenix.Socket
  use Absinthe.Phoenix.Socket, schema: MyAppWeb.Schema

  # ...
end

This adds a special channel to the socket - Absinthe.Phoenix.Channel - that handles communication with the Absinthe.Subscription server we added to our supervision tree.

I know this is a lot to take in, but Absinthe does a really good job of managing everything internally. We only need the few lines of code we see above for everything to work on the server side.

In the previous post of this series, we saw that we could add a context available to all our queries and mutations. Since the new requests for subscriptions are now going through the UserSocket instead of the regular:graphql pipeline in the schema, we don't have access to the context inside the subscriptions. To get it back, we need to put it inside the socket during the connect callback:

defmodule MyAppWeb.UserSocket do
  use Phoenix.Socket
  use Absinthe.Phoenix.Socket, schema: MyAppWeb.Schema

  # ...

  @impl true
  def connect(query_params, socket, _connect_info) do
    case authorize_user(socket, query_params) do
      {:ok, user} ->
        {:ok, Absinthe.Phoenix.Socket.put_options(socket, context: %{current_user: user})}
      {:error, _reason} ->
        # Reject the connection
        :error
    end
  end

  defp authorize_user(socket, query_params) do
    # Fetch the user here (for example, from a token in the query_params)
    {:ok, user}
  end
end

Client-Side Setup with Apollo

Now that our server is ready to accept connections over WebSocket, let’s set up the client. Since Apollo is usually the go-to client for client-side GQL usage, I will only cover that in this post. If you are using Relay, check out Using Absinthe with Relay. For usage without external clients, check out the @absinthe/socket package, which provides a way to send subscription requests and observe the results.

If you are new to Apollo, I suggest going through the Introduction to Apollo Client docs before going further because we will use advanced concepts to customize our client.

In a regular setup, Apollo is usually configured to use an HTTP Link that, by default, sends a POST request to the endpoint whenever a new query/mutation is executed. We now want to modify it to send the request over a WebSocket connection instead of a regular HTTP request.

First, let’s create a standard Phoenix WebSocket connection:

// assets/js/app.js
import { Socket as PhoenixSocket } from "phoenix";

const phoenixSocket = new PhoenixSocket("ws://localhost:4000/socket", {
  params: () => {
    const token = "xxx"; // an auth token (e.g. from cookies or another authentication process)
    return { token };
  },
});

We can pass any hash as params when opening the socket connection. The above is just an example where we send a token that can then be used inside MyAppWeb.UserSocket.connect/3 above to authorize a user.

Next, wrap it inside an AbsintheSocket and create a link for use with the Apollo client:

import * as AbsintheSocket from "@absinthe/socket";
import { createAbsintheSocketLink } from "@absinthe/socket-apollo-link";

const absintheSocket = AbsintheSocket.create(phoenixSocket);
const websocketLink = createAbsintheSocketLink(absintheSocket);

Note that you will need to install @absinthe/socket and @absinthe/socket-apollo-link with npm/yarn.

Finally, we will set up the Apollo client to use this link:

import { ApolloClient, InMemoryCache } from "@apollo/client";

const client = new ApolloClient({
  link: websocketLink,
  cache: new InMemoryCache(),
});

While this approach works for all types of documents (queries, mutations, or subscriptions), it sends all of them over the WebSocket. It is usually better to send regular POST requests to the API endpoint in your application for regular queries and mutations and only use WebSocket when creating subscriptions.

This is possible using the Apollo client’s split function to choose which ApolloLink to use for each request. Let’s set it up to only use the websocketLink for subscriptions and create an HttpLink for other operations:

import { Socket as PhoenixSocket } from "phoenix";
import * as AbsintheSocket from "@absinthe/socket";
import { createAbsintheSocketLink } from "@absinthe/socket-apollo-link";
import { ApolloClient, HttpLink, InMemoryCache, split } from "@apollo/client";
import { hasSubscription } from "@jumpn/utils-graphql";

// Create the websocket link
const phoenixSocket = new PhoenixSocket("ws://localhost:4000/socket", {
  params: () => {
    const token = "xxx"; // an auth token (e.g. from cookies or another authentication process)
    return { token };
  },
});
const absintheSocket = AbsintheSocket.create(phoenixSocket);
const websocketLink = createAbsintheSocketLink(absintheSocket);

// Create http link
const httpLink = new HttpLink({ uri: "http://localhost:4000/api" });

// Create a split link
const link = split(
  (operation) => hasSubscription(operation.query),
  websocketLink,
  httpLink,
);

// Create the client
const client = new ApolloClient({
  link,
  cache: new InMemoryCache(),
});

Here we are using the hasSubscription helper method from @jumpn/utils-graphql to check if a query has a subscription. If so, we route it to the websocketLink and all other queries to httpLink.

Finally, with this setup, you can start using useSubscription inside your react component or client.subscribe outside React components to subscribe for updates. Let's see how to create a React component that shows new comments as they are posted:

// assets/js/components/NewComments.jsx

import { gql, useSubscription } from "@apollo/client";
import { useState } from "react";

const COMMENTS_SUBSCRIPTION = gql`
  subscription ($postId: ID!) {
    commentAdded(postId: $postId) {
      id
      body
    }
  }
`;

export default function NewComments({ postId }) {
  const [comments, setComments] = useState([]);

  useSubscription(COMMENTS_SUBSCRIPTION, {
    variables: { postId },
    onData: ({ commentAdded }) => {
      setComments((existing) => [...existing, commentAdded]);
    },
  });

  return (
    <div>
      {comments.map((comment) => (
        <div key={comment.id}>{comment.body}</div>
      ))}
    </div>
  );
}

And that's it!

Wrap Up

GraphQL subscriptions provide a great way to add real-time behavior to your app. In this post, we saw how to set up the server and the client to support subscriptions.

Since plain HTTP isn’t well suited for long-running connections with two-way server communication, we also saw how to allow Absinthe on the server and Apollo client to communicate over WebSockets for subscriptions.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Add a Form to a Modal in Phoenix 1.7

Roy Tomeij — 2023-08-01T00:00:00+00:00

In part one of this series, we introduced the core generated components when bootstrapping a new Phoenix project. We used a button and a modal from the core components to lay the groundwork for a "create modal".

In this post, we will put a form onto the modal and create pets.

Let's get started!

Note: As in the last post, you can follow along with our companion repo.

Adding a Form to Our Phoenix Application

There is a simple form included in PetacularWeb.CoreComponents. This expects two slots: an :inner_block, which will be our form input components, and :actions, which will be the submit buttons. The :actions slot is a named slot, meaning, we can render the components we want to use as buttons inside of an <:actions> tag, like this:

# in /lib/petacular_web/pages/home_live.ex

<PetacularWeb.CoreComponents.simple_form for={@create_form} phx-submit="create_pet">
  <:actions>
    <PetacularWeb.CoreComponents.button>
      Save
    </PetacularWeb.CoreComponents.button>
  </:actions>
</PetacularWeb.CoreComponents.simple_form>

The input components are also in PetacularWeb.CoreComponents and produce the correct kind of input based on the attrs used to define them. We just need a text field for now, so we can define the text input like so:

<PetacularWeb.CoreComponents.input field={@create_form[:name]} label="Name" />

If you don't define a type attr, it defaults to a text field. The @create_form is created from a changeset (as we will see in a moment).

Phoenix's Component `to_form`

Let's take a moment to talk about forms and changesets. We used to pass changesets directly to forms. But in Phoenix 1.7, a new to_form function generates a Phoenix.HTML.Form struct from the given changeset.

This seemingly small change is a massive improvement, so it's worth talking about. First, it provides us with some indirection. Calling to_form ensures that what the <.form component sees is a Phoenix.HTML.Form struct, rather than a changeset. That means if we wanted to swap away from a changeset and use something else in the future (like a bare map of params), we could — with no changes to the <.form component itself. We just change the places we call to_form in mount and handle_event.

Next, the Phoenix.HTML.Form handles change tracking better. Up until Phoenix 1.7, if a form field changed, the whole form was re-rendered. Usually that's fine, but if you have a large complex form or dropdowns with lots of options, it can be a big boon not to have to re-render all of that each time you change a field.

Finally, it also simplifies the syntax. When we passed changesets through to forms, we used the :let attribute to assign the changeset to a variable we could refer to. Now we can omit that entirely. We can also refer to the value of a form field more simply; previously, you would have done something like: value={Ecto.Changeset.fetch_field!(@changeset, :my_field)}. With the form struct, it becomes: value={@form[:my_field].value}

A full before and after might look like this.

Before:

@impl true
def mount(_params, _session, socket) do
  default_assigns = %{
    changeset: Petacular.Pet.create_changeset(%{})
  }

  {:ok, assign(socket, default_assigns)}
end

~H"""
<.form :let={f} for={@changeset} phx-submit="...">
  <%= <Phoenix.HTML.Form.text_input(f, :my_field, value: Ecto.Changeset.fetch_field!(@changeset, :my_field)) %>
</.form>
"""

And after:

@impl true
def mount(_params, _session, socket) do
  default_assigns = %{
    create_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
  }

  {:ok, assign(socket, default_assigns)}
end

~H"""
<.form for={@create_form} phx-submit="...">
  <input type="text" name={@create_form[:my_field]} value={@create_form[:my_field].value} />
</.form>
"""

Note how we don't need HTML.text_input anymore, and we don't have to reach into the changeset.

Check out the official Phoenix docs for more information.

Using `to_form`

First, set up a form in mount:

@impl true
def mount(_params, _session, socket) do
  default_assigns = %{
    create_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
  }

  {:ok, assign(socket, default_assigns)}
end

Now we can build a form and use the pre-built inputs. These accept the field from the form (which is as easy as field={@create_form[:name]}).

<PetacularWeb.CoreComponents.modal id="create_modal">
  <h2>Add a pet.</h2>

  <PetacularWeb.CoreComponents.simple_form
    for={@create_form}
    phx-submit="create_pet"
  >
    <PetacularWeb.CoreComponents.input field={@create_form[:name]} label="Name" />
    <:actions>
      <PetacularWeb.CoreComponents.button>
        Save
      </PetacularWeb.CoreComponents.button>
    </:actions>
  </PetacularWeb.CoreComponents.simple_form>
</PetacularWeb.CoreComponents.modal>

Saving the Form

If we save and refresh the form, we should see that we can click a button and enter a name into it. The final thing to do to get this working is to write an event handler for the form submission.

Here is the first gotcha. We are using changesets to validate our forms. Because live views are stateful, we might be tempted to re-use the changeset in our assigns when we submit the form. This is not a good idea. What's worse, Ecto.Changeset.cast et al. will happily accept a changeset as input, meaning it's a very easy trap to fall into without realizing you are doing something wrong. The symptom might be errors in your changeset not disappearing as you expect, for example.

This trap is harder to fall into if you use to_form because the changeset is not in the assigns — the form struct is. This is another good reason to use to_form!

So every time we want to cast some params, we must create a fresh changeset. Let's do that:

# in lib/petacular_web/pages/home_live.ex
@impl true
def handle_event("create_pet", %{"pet" => params}, socket) do
  case Repo.insert(Petaclular.Pet.create_changeset(params)) do
    {:error, message} ->
      {:noreply, socket |> put_flash(:error, inspect(message))}

    {:ok, _} ->
      new_assigns = %{}
      {:noreply, assign(socket, new_assigns)}
  end
end

In the case of an error, we add a flash that shows it to the user, and when we successfully complete, we need to do two things:

Show the created pet.
Close the modal.

To show the new pet, we first need to render all pets on the page. So we will fetch them all from the DB in mount and then add some code to render them:

# in lib/petacular_web/pages/home_live.ex
@impl true
def mount(_params, _session, socket) do
  default_assigns = %{
    pets: Repo.all(Petacular.Pet),
    create_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
  }

  {:ok, assign(socket, default_assigns)}
end

...

<h1 class="font-semibold text-3xl mb-4">Pets</h1>

<div class="w-50 mb-4">
  <%= for pet <- @pets do %>
    <p>Name: <span class="font-semibold"><%= pet.name %></span></p>
  <% end %>
</div>

Now that we can see them, we can refresh the list of pets when we add one successfully:

# in lib/petacular_web/pages/home_live.ex
@impl true
def handle_event("create_pet", %{"pet" => params}, socket) do
  case Repo.insert(Petacular.Pet.create_changeset(params)) do
    {:error, message} ->
      {:noreply, socket |> put_flash(:error, inspect(message))}

    {:ok, _} ->
      new_assigns = %{
        pets: Repo.all(Petacular.Pet)
      #  ^^^ add new
      }

      {:noreply, assign(socket, new_assigns)}
  end
end

This is great. If we try adding a pet, we will see the pet gets created and appears on the page, but the modal remains open.

Closing the Modal with `push_event`

Once we have successfully added a pet, we want to close the modal. There is already a "close the modal" button on the modal; the little X in the top right-hand corner. What if we could just target that?

Well, we can! There is a function called push_event that lets us emit a JS event from the backend. We can add some JavaScript that listens for that event and triggers a "click" event for the close button, effectively closing the modal.

First, we call push_event in the event handler:

# in lib/petacular_web/pages/home_live.ex
@impl true
def handle_event("create_pet", %{"pet" => params}, socket) do
  case Repo.insert(Petacular.Pet.create_changeset(params)) do
    {:error, message} ->
      {:noreply, socket |> put_flash(:error, inspect(message))}

    {:ok, _} ->
      new_assigns = %{
        pets: Repo.all(Petacular.Pet),
        create_form: Phoenix.Component.to_form(Petacular.Pet.create_changeset(%{}))
        # reset the form back to being empty ^^^
      }

      socket =
        socket
        |> assign(new_assigns)
        |> push_event("close_modal", %{to: "#close_modal_btn_create_modal"})
        #  ^^^^ add this
      {:noreply, socket}
  end
end

Now to react to the event, we have two options:

Add a global event listener for the event.
Use a hook.

Let's try the second approach. First, we'll wire up the hook in app.js:

const ModalCloser = {
  mounted() {
    this.handleEvent("close_modal", () => {
      this.el.dispatchEvent(new Event("click", { bubbles: true }));
    });
  },
};

let liveSocket = new LiveSocket("/live", Socket, {
  params: {
    _csrf_token: csrfToken,
  },
  hooks: {
    ModalCloser: ModalCloser,
  },
});

Now let's put the hook onto the close modal button in PetacularWeb.CoreComponents (remember, anything that has a hook also needs an ID). To help us later cater for multiple modals on one page, let's use the required @id attr as part of the id:

<button
  id={"close_modal_btn_" <> @id}
  phx-hook="ModalCloser"
  <!-- ^^ Add these ^^ -->
  phx-click={JS.exec("data-cancel", to: "##{@id}")}
  type="button"
  class="-m-3 flex-none p-3 opacity-20 hover:opacity-40"
  aria-label={gettext("close")}
>
  <.icon name="hero-x-mark-solid" class="h-5 w-5" />
</button>

You can see all the changes in this commit. Fantastic, we now have a working create modal! 🎉

Wrapping Up

In the second part of this series, we successfully added a form to the modal that creates new pets.

Stay tuned for the third and final part, where we will edit an existing pet.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Observe Your Phoenix App with Structured Logging

Roy Tomeij — 2023-07-18T00:00:00+00:00

In this post, we'll configure a Phoenix LiveView application to use a structured logger. We'll then use AppSignal to correlate log events with other telemetry signals, like exception reports and traces.

Along the way, you'll learn about the benefits of structured logging, and you'll see how to configure a distinct framework and application logger in your Phoenix app.

Let's get started!

What is Structured Logging?

Structured logs conform to a specific structure in which an agreed-upon set of key/value pairs describe common log attributes. Without structured logging, you might get a log line that looks something like this:

20:34:05.879 request_id=F1ssWNn62XasBTYAAAEj [info] GET /
20:34:05.911 request_id=F1ssWNn62XasBTYAAAEj [info] hello from the page controller
20:34:05.919 request_id=F1ssWNn62XasBTYAAAEj [info] Sent 200 in 39ms

A few bits of information make up each log statement:

a timestamp
a level (in this case, info)
a log message
a bit of metadata (the request_id)

In this unstructured format, all the data is included in one string, so it's difficult to search and analyze log statements based on each data point.

Structured logging solves this problem for us. It breaks each piece of information into its own key/value pair. Here's an example of these same log statements structured using the logfmt format:

timestamp="20:38:15.973 2023-05-01" level=info message="GET /" request_id=F1sskxS1k-w0jUIAAAKB
timestamp="20:38:16.013 2023-05-01" level=info message="hello from the page controller" request_id=F1sskxS1k-w0jUIAAAKB
timestamp="20:38:16.031 2023-05-01" level=info message="Sent 200 in 58ms" request_id=F1sskxS1k-w0jUIAAAKB

Now you can search and analyze log statements based on the timestamp, level, message, and request_id fields. For example, you could search all log statements with a shared request_id, filter log lines by their level key, or search logs within a range of timestamps.

Now that we can see the benefits of using structured logs, we can configure our Phoenix LiveView application to use the logfmt format in development and production. But first, we'll take a closer look at how Phoenix apps configure their loggers out of the box.

Logger Configuration in Phoenix

By default, your Phoenix application will configure its logger to output unstructured logs to the console. Open up your config.exs file, and you'll see something like this:

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]

This means the logger will use the console backend to print log messages to the console using the specified (unstructured) format. This configuration says that only the request_id from the default metadata list should be included in each log statement. Elixir's Logger adds a certain set of metadata to all log statements, and you can find the full list here. The request_id metadata, however, is not added by default with the Elixir Logger. Instead, it is added by the Plug.RequestId plug that you'll see added to your application's endpoint.ex file like this:

# endpoint.ex
plug Plug.RequestId

Let's take a quick peek at the rest of the default metadata by setting metadata: :all in config.exs:

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: :all

Now, when we fire up our app and visit the root path, we'll see something like this:

21:09:24.947 erl_level=info application=phoenix domain=elixir file=lib/phoenix/logger.ex function=phoenix_endpoint_stop/4 line=231 mfa=Phoenix.Logger.phoenix_endpoint_stop/4 module=Phoenix.Logger pid=<0.815.0> request_id=F1suRjwyEp4CjWAAAACi [info] Sent 200 in 2ms

That's a lot of information! Too much information for our development environment. These logs are so information-dense that they're difficult to parse with our human eyes. For that reason, you'll notice that the dev.exs file overwrites this configuration to simplify it:

config :logger, :console,
  format: "$metadata[$level] $message\n"

Here, the timestamp and metadata are excluded entirely, except for the metadata's $level data point.

Now that we've seen where and how your Phoenix app configures the logger out-of-the-box, let's configure the logger to emit structured logs that conform to logfmt.

Using Logfmt in Phoenix Development

First up, we'll configure our development environment to use the console logger backend with a logfmt formatter. For this, we need the LogfmtEx dependency added to our application. Open up your mix.exs file and add it like this:

{:logfmt_ex, "~> 0.4"}

Go ahead and run mix deps.get to install the new dependency.

Next up, we'll configure our application's console logger to use this formatter. In dev.exs, add the following:

config :logger, :console, format: {LogfmtEx, :format}

Now, when you start up your application, you should see structured logs like this:

timestamp="21:18:36.828 2023-05-01" level=info message="Sent 200 in 45ms" request_id=F1suxrg8tRisBTYAAAmh

Notice that we haven't overwritten the :metadata key from the console backend config in config.exs, so we're still only logging the request_id data point from the metadata.

Now we've successfully emitted structured logs to the console, but this isn't good enough for production. To operate a Phoenix application in production, we need to emit our logs to a third-party service that allows us to search, aggregate, and analyze them at scale. This is where AppSignal comes in.

In the next section, we'll incorporate the AppSignal logger into our Phoenix application and configure it to use logfmt.

Using Logfmt with the AppSignal Logger in Production

If you don't already have AppSignal installed and configured in your application, you can follow the instructions here. Assuming you have an AppSignal account and organization configured, we're ready to set up the AppSignal logger backend by following the steps in AppSignal's Logging from Elixir docs. We'll use the Elixir integration rather than set up a custom source.

In our runtime.exs file, we'll specify the Appsignal.Logger.Backend, like this:

config :logger, :backends, [
  {Appsignal.Logger.Backend, [group: "phoenix", format: "logfmt"]}
]

Also, in runtime.exs, you'll want to set the application's log level to :info, to ensure we send all logs at the info level and below to AppSignal:

config :logger, level: :info

This will forward our application's logs to our AppSignal account and group them under the "phoenix" group designation. It will treat the format of the log statements as logfmt and allow us to search and analyze our logs based on their attributes.

If you log into your AppSignal account, select your organization, and select "Logging" -> "Application", you'll see something like this:

Note that the log line includes the timestamp, hostname, level, and group (in this case, "phoenix"). Since we've specified "logfmt" for the Appsignal.Logger.Backend, each of these attributes is indexed and searchable. All we have to do is double-click any of these attributes on the log line itself, and the search filter on the top of the page will update and execute:

Let's take a closer look at one of these log lines now. The "hello from the page controller" log line is emitted explicitly in application code by the following line added to the PagesController:

# pages_controller.ex
require Logger

def home(conn, _params) do
  Logger.info("hello from the page controller")
  render(conn, :home, layout: false)
end

If we click the "+" icon next to the log line, we see all of the log line's metadata:

You can see that the Appsignal.Logger.Backend emits all of the default metadata, along with the request_id. Since we're using logfmt, each data point is searchable. So, just like we did with the log attributes displayed in the log line itself, we can click on any metadata attributes to get them added to the search. If you click on the request_id_string attribute, you'll see the search query update to look like this:

In fact, we can add additional metadata when we emit log lines (using calls to Logger), like this:

Logger.info("hello from the page controller", %{user_id: conn.assigns.current_user.id})

With that, the user_id attribute is displayed in the details of each log line and searchable just like the request_id metadata.

So far, we've:

Configured our Phoenix application's logger to use the Appsignal.Logger.Backend.
Taught that backend to use logfmt.
Seen logs emitted and consumed into AppSignal, and got a feel for the basic logging search functionality there.

Before we continue exploring the AppSignal logging UI and some of its more useful features, we need to address a problem that's cropped up.

Framework Vs. Application Logs

Taking another look at the log statements our Phoenix app has emitted to AppSignal so far, we see something like this:

05-01-2023 10:21:04.383[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Access StreamChatWeb.Endpoint at http://localhost:4000
+05-01-2023 10:21:04.363[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Running StreamChatWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:4000 (http)
+05-01-2023 10:19:02.709[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]GET /asdfads
+05-01-2023 10:18:51.429[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 4ms
+05-01-2023 10:18:51.426[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]GET /
+05-01-2023 10:18:51.388[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 302 in 252ms
+05-01-2023 10:18:51.137[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]POST /users/log_in
+05-01-2023 10:18:50.201[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]CONNECTED TO Phoenix.LiveView.Socket in 16µs Transport: :websocket Serializer: Phoenix.Socket.V2.JSONSerializer Parameters: %{"_csrf_token" => "UQshLhEWIzUbAXgLQTE5bFNXRW8LZ00w3xVHuWYmHt1avWPZ4e6-97ut", "_live_referer" => "undefined", "_mounts" => "0", "_track_static" => %{"0" => "http://localhost:4000/assets/app.css", "1" => "http://localhost:4000/assets/app.js"}, "vsn" => "2.0.0"}
+05-01-2023 10:18:50.130[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 34ms
+05-01-2023 10:18:50.097[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]GET /users/log_in
+05-01-2023 10:18:48.255[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 689µs
+05-01-2023 10:18:48.254[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]GET /
05-01-2023 10:18:48.187[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]hello from the page contoller
+05-01-2023 10:18:48.215[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 10ms
+05-01-2023 10:18:48.206[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]POST /users/log_out
05-01-2023 10:19:25.413[application][ERROR][SOPHIEs-MacBook-Pro-2.local][phoenix]** (RuntimeError) boom! iex:5: (file) (elixir 1.14.2) src/elixir.erl:309: anonymous fn/4 in :elixir.eval_external_handler/1 (stdlib 4.1.1) erl_eval.erl:748: :erl_eval.do_apply/7 (stdlib 4.1.1) erl_eval.erl:987: :erl_eval.try_clauses/10 (elixir 1.14.2) src/elixir.erl:294: :elixir.eval_forms/4 (elixir 1.14.2) lib/module/parallel_checker.ex:107: Module.ParallelChecker.verify/1 (iex 1.14.2) lib/iex/evaluator.ex:329: IEx.Evaluator.eval_and_inspect/3 (iex 1.14.2) lib/iex/evaluator.ex:303: IEx.Evaluator.eval_and_inspect_parsed/3 (iex 1.14.2) lib/iex/evaluator.ex:292: IEx.Evaluator.parse_eval_inspect/3 (iex 1.14.2) lib/iex/evaluator.ex:187: IEx.Evaluator.loop/1 (iex 1.14.2) lib/iex/evaluator.ex:32: IEx.Evaluator.init/4
+05-01-2023 10:18:47.578[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 7ms
+05-01-2023 10:18:47.572[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]GET /
+05-01-2023 10:18:46.901[application][INFO][SOPHIEs-MacBook-Pro-2.local][phoenix]Sent 200 in 52ms

There is a lot of noise in here mixed with some useful bits of information. You can see one error log that will very likely be helpful in debugging, as well as one instance of our "hello from the page controller" log statement. Those two log lines are very hard to detect amidst the many log lines emitted that indicate the server is up, the WebSocket channel is connected, certain web requests are served, and so on.

These noisy logs are emitted not from our application code, but from the Phoenix framework itself. They don't tell us anything interesting, and they are not an essential part of the overall observability picture of our application. Instead, they create noise and contribute to high log volume, making it harder to find and identify the log statements that we intend to emit from application code.

For this reason, framework logs should be emitted at the FATAL level. Application logs that are emitted when exceptions are raised or by explicit calls to Logger should log at the INFO level. Phoenix doesn't expose an easy way for us to do that, but we can build our own custom logger backend to solve this problem. Let's do that now.

Build A Custom Logger Backend

A logger backend is really nothing more than a GenServer, and must conform to the following API:

Implement an init/1 function
Implement a handle_event/3 function

Here's the skeleton for our backend:

# lib/stream_chat/logger/backend.ex
def init({__MODULE__, options}) do
    {:ok, Keyword.merge([group: __MODULE__, format: :plaintext], options)}
  end

  def handle_event(
        {level, _gl, {Logger, message, _timestamp, metadata}},
        options
      ) do
      # ...
  end
end

The init/1 function will be called with whatever config we pass to the call to configure the logger backend in runtime.exs. It returns the {:ok, state} tuple.

The handle_event/2 function will be called with all the data we need to emit a log statement with the Appsignal.Logger, including the:

Level
Log message
Metadata
Internal state of the logger backend process that we established in init/1

Whenever the Phoenix framework itself calls the Phoenix application's logger, the metadata variable passed to handle_event/2 will include an :application key pointing to a value of :phoenix. Where logs are emitted by an explicit call to Logger or by an exception raised in our application, the metadata keyword list will not include an :application key. We want to be able to write some code that says: "Only log statements coming from Phoenix if they are at the configured log level or below". We can do exactly that by adding some logic that looks for metadata with application: :phoenix.

Before we write that code, let's take a look at how we will configure this custom logger backend in our runtime.exs file:

config :logger, :backends, [
  {StreamChat.Logger.Backend,
   [
     format: "logfmt",
     application_config: [
       phoenix: [
         level_lower_than: :fatal
       ],
       default: [
        level_lower_than: :info
       ]
     ]
   ]}
]

Here, we include a key :application_config, which points to another keyword list. That keyword list contains configs for the :phoenix app along with the :default app. This data will be passed to the init/1 function and stored in our backend's state. So, when handle_event/2 is called later on, we can do the following:

Check if the metadata contains application: :phoenix.
If so, check the provided level and only log if it is at or below the level stored in application_config: [phoenix: [level_lower_than: level]] in state.

Here's what our function will look like:

 @levels %{
    fatal: 0,
    error: 1,
    warn: 2,
    info: 3,
    debug: 4,
    trace: 5
  }

def handle_event(
      {level, _gl, {Logger, message, _timestamp, metadata}},
      options
    ) do
  application = metadata[:application] || :default

  log_event(
    @levels[level],
    @levels[options[:application_config][application][:level_lower_than]],
    message,
    metadata,
    options
  )

  {:ok, options}
end

def log_event(level, level_lower_than, message, metadata, options)
    when level <= level_lower_than do
  Appsignal.Logger.log(
    level,
    to_string(metadata[:group] || options[:group]),
    IO.chardata_to_string(message),
    Enum.into(metadata, %{}),
    options[:format]
  )
end

def log_event(_level, _level_lower_than, _message, _metadata, _options),
  do: :noop

We pull the application config out of state and pass that along with the provided level to the log_event helper function. That function includes a guard clause that will cause it to no-op if the provided level is higher than the level configured in runtime.exs for the given application. If we should in fact log, we do so by calling directly on the AppSignal.Logger.log function. This allows us to emit logs to AppSignal at a specified level, for a given group, with a message and any metadata.

With this, we will suppress Phoenix framework logs above the FATAL level, but ensure that logs emitted by application code are emitted at INFO.

If you're hesitant to suppress framework logs in this way, remember that logs are just one piece of the observability puzzle. Rather than recording every instance of a GET / request, we are more likely to care about trends in serving such requests over time. We want the ability to drill down into the details of some such requests, especially where they are anomalous (for example, if a request experiences a high degree of latency).

This is where traces come in, along with other telemetry data, like metrics and exceptions.

In the next section, we'll look at how you can use AppSignal to correlate logs with traces and other types of telemetry, giving you a full picture of how your application is performing.

Correlating Log Events and Other Telemetry with AppSignal

Application logs are just one part of your overall observability posture. They should be used to record specific events of interest in your application's lifecycle and provide additional details and context in the event of an error or exception.

You should leverage application logs alongside other telemetry data like metrics, traces, and exception reports to fully observe your application in production.

Here's a very high-level overview of how you can view each type of telemetry data that your application emits:

Logs record that an event occurred (for example, a failed user log-in attempt).
Metrics record trends in event occurrences over time. For example, you can record the distribution of request latency to a particular endpoint, or monitor the capacity utilization of your system.
Traces record details of the lifecycle for a particular piece of code flow. For example, drill down into the trace of a specific request to understand possible causes of latency.
Exception reports show when an unexpected or "exceptional" error occurred. If your code encounters an exception, you can see the details of that exception, including the backtrace and full error message.

These telemetry signals become especially powerful when they can be correlated. For example, when viewing a specific exception report, you may want to gain more insight into the performance of your overall system at the time the exception occurred. In this case, the ability to view logs and/or traces from that same timeframe can provide you with the extra information you need to remedy bugs and improve overall application performance.

Monitoring Your Logs Using AppSignal

Correlating telemetry data can be a particularly hard problem to solve in the observability domain, especially when using different third-party services for things like logging, tracing, and exception reporting. AppSignal provides functionality for managing all of these types of telemetry and empowers you to correlate different types of telemetry data to get the most out of various signals. Let's take a look at an example now.

We'll use the "Errors" -> "Issue list" as the entry point for this exercise. I've selected an exception report from this list, displayed here:

We can correlate this exception report with some traces emitted by the live view that encountered this error. Note that I've configured our application to emit LiveView traces to AppSignal by following the instructions here. By clicking on "samples" and selecting an exception sample to examine, we can see the following exception details:

Now, in addition to the original exception report, we can see other useful information like the event that the live view was trying to handle when the exception occurred and the payload of that event, in addition to the full exception backtrace.

Finally, we can use the Time Detective feature to explore the full range of telemetry data, including traces, that our application emitted around the time of this exception. If we click on the "Time Detective" action here:

We'll see something like this:

The "Performance" view brings up some trace samples from the time period that we can drill down into. Here, we can see a list of traces that occurred around the time of the incident, including traces for the various LiveView lifecycle callbacks. From the details in the exception sample, we know this error occurred when an event handler in our live view tried to handle the "update" message. So, let's select the handle_event trace. We'll see something like this:

You can also select the "Logs" tab to view logs from this time period.

The Time Detective tool can be accessed via the logging UI as well. Just expand a given log line and select "Time Detective" from the bottom of the expanded view:

Then you'll have access to all of the telemetry data that can be correlated to your log statement based on timeframe. This could be especially useful if you see a high volume of a particular request being logged, or note an interesting error log. For any interesting or informative log statements, a deep dive into correlated telemetry data is just a click away.

Logging in Phoenix: A Few Final Tips

You're now ready to leverage logging as one part of the overall observability story in your Phoenix application. Just a few guiding rules to keep in mind when you do:

Prefer structured over unstructured logs so that you can query and analyze your application's logs with tools like AppSignal.
Suppress framework logs to the FATAL level and emit application logs at INFO.
Treat logs as just one part of your observability posture. Logs are most useful when correlated with other telemetry data, like exceptions and traces.

Wrap Up

We've covered a lot of ground in this post, so let's wrap up before you go.

We discussed the benefit of structured over unstructured logging, and configured our Phoenix application to use logfmt with the console logger backend. We then saw the benefits of structured logging in action in AppSignal.

Finally, we took our log analysis skills to the next level when we suppressed framework logs and explored how our meaningful application logs correlated with other telemetry data in AppSignal.

Now take on the world of Phoenix logging. Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Deep Dive into Mutations with Absinthe

Roy Tomeij — 2023-07-04T00:00:00+00:00

In the last two posts of this series, we saw how easy it is to integrate Absinthe into your app to provide a GraphQL API for querying data. We also shared some useful tips for building and maintaining large schemas and optimizing queries.

This post will show how we can provide an API to create GraphQL mutations with Absinthe for Elixir.

Let's get started!

A Note on Queries Vs. Mutations in GraphQL

GraphQL places a clear distinction between queries and mutations. Technically, we can implement any query to write data to the server. But GraphQL convention clearly separates them into different top-level mutation types.

Mutations in GraphQL

First, let’s see what a typical mutation looks like:

mutation CreatePost($post: PostCreateInput!) {
  createPost(post: $post) {
    post {
      id
    }
    errors
  }
}

This is a mutation to create a new post. It accepts a variable named post of type PostCreateInput (which is required because ! follows the type name). This variable is passed into the createPost field inside the top-level mutation type.

The result of that mutation is a post (which can have further selections) and an errors array. Note that we have named the mutation CreatePost but that is just a client-side identifier. Here is what a sample response from this mutation looks like:

{
  "data": {
    "createPost": {
      "post": {
        "id": "1"
      },
      "errors": null
    }
  }
}

Input Objects with Absinthe for Elixir

Let’s see how we can implement this with Absinthe. The first thing we need is an input_object that can be used as a variable for the mutation. Let’s create one now:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  enum :post_state_enum do
    value :active
    value :draft
  end

  input_object :post_create_input do
    field :title, non_null(:string)
    field :author_id, non_null(:id)
    field :body, non_null(:string)
    field :state, :post_state_enum, default_value: :draft
  end
end

Here, we use the enum macro to create an enum named post_state_enum with two possible values — draft and active.

We then use the input_object macro to define an input object named post_create_input which has four fields. Note that the definition of fields changes slightly from what we would use in an output type. For example, for fields inside an input object, we can add a default_value to a non-required field. Absinthe will fill the field in if the user doesn’t provide a value.

Defining Mutations in Absinthe

Next, let’s define our mutation type in the schema. All individual mutations (like createPost) will be fields inside this mutation:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  mutation do
    field :create_post, non_null(:post_mutation_result) do
      arg :post, non_null(:post_create_input)
    end
  end
end

Here, we use the mutation..do..end block to define the base mutation type. And inside it, we define a field named create_post. This takes a single argument named post of the post_create_input type defined above. The result of this field is of type post_mutation_result. We haven’t defined it yet, so let’s do that now.

object :post_mutation_result do
  field :post, :post
  field :errors, list_of(non_null(:string))
end

Now on to the interesting part — actually performing the mutation operation. This is similar to what we already did for queries. We will define a resolver to handle it.

If you remember from our previous post, a 3-arity resolver receives the parent object, the attributes, and an Absinthe.Resolution struct. Let’s define that first to create the post:

defmodule MyAppWeb.Resolvers.PostResolver do
  def create_post(_parent, %{post: attrs}, _resolution) do
    case MyAppWeb.Blog.create_post(attrs) do
      {:ok, post} ->
        {:ok, %{post: post}}
      {:error, changeset} ->
        {:ok, %{errors: translate_changeset_errors(changeset)}}
    end
  end
end

In the resolver, we use the post attributes to create a post. If that is successful, we respond with the post object (the result map should correspond to the mutation's result type — post_mutation_result, in our case). On an error, we respond with the errors. Note that translate_changeset_errors is a custom helper function that translates Ecto.Changeset errors into an array of strings.

With the resolver in place, we can now plug it into our schema to create a post:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  mutation do
    field :create_post, :post_mutation_result do
      arg :post, non_null(:post_create_input)

      resolve &MyAppWeb.Resolvers.PostResolver.create_post/3
    end
  end
end

Now, when we execute the above mutation against our schema, Absinthe will call the create_post function. The return value from the resolver's {:ok, value} tuple will then be used as the mutation's result and returned to the user.

Authorization in Your Elixir App with GraphQL and Absinthe

Up until now, we have been discussing everything as if we're using an open public API. But in most apps, this is not how things work. Often, certain APIs are only available to logged-in users, and this is even more important in the case of mutations. Let’s see how we can add authorization to our API.

First, we will need to identify the users making requests. In the first post of this series, we used Absinthe.Plug to handle all the requests coming into the /api endpoint using the MyAppWeb.Schema schema. This doesn’t handle any user authorization for us.

Absinthe provides a context concept containing shared information that might be useful for all queries and mutations. This is passed to all resolver functions that accept an Absinthe.Resolution struct inside the context field.

Let’s fix that first to identify users before a request is forwarded to Absinthe.

An Example Using Absinthe Context

Update the router in your app to execute an additional plug before forwarding a request:

defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  pipeline :graphql do
    plug MyAppWeb.Schema.Context
  end

  scope "/" do
    pipe_through [:api, :graphql]

    forward "/api", Absinthe.Plug, schema: MyAppWeb.Schema
  end
end

We've added a new MyAppWeb.Schema.Context plug to the requests. Let’s implement it to put a user in the Absinthe context.

defmodule MyAppWeb.Schema.Context do
  @behaviour Plug

  import Plug.Conn

  #...

  def call(conn, _default) do
    Absinthe.Plug.put_options(conn, context: absinthe_context(conn))
  end

  def absinthe_context(conn) do
    %{conn: fetch_query_params(conn)}
    |> put_user()
  end
end

Note: I have intentionally left out some app-specific parts of the plug. Here is the full code of that module if you are interested.

An interesting part of the code in the above code block is the use of Absinthe.Plug.put_options/2 to set values that Absinthe.Plug will pass to Absinthe when executing the query/mutation. This is similar to how we use Plug.Conn.assign to assign something on the conn.

Internally, Absinthe.Plug puts a private assign inside the conn and passes it as the third parameter to Absinthe.run/3 when executing the document. The context option we use above is a special value that Absinthe will then pass to all resolvers inside the Absinthe.Resolution struct.

Now let’s update our create_post/3 resolver function to use this context and deny the request if the user isn’t present.

defmodule MyAppWeb.Resolvers.PostResolver do
  def create_post(
    _parent,
    %{post: attrs},
    %Absinthe.Resolution{context: %{user: nil}}
  ) do
    {:error, "unauthorized"}
  end

  def create_post(_parent, %{post: attrs}, %Absinthe.Resolution{}) do
    case MyAppWeb.Blog.create_post(attrs) do
      {:ok, post} ->
        {:ok, %{post: post}}
      {:error, changeset} ->
        {:ok, %{errors: translate_changeset_errors(changeset)}}
    end
  end
end

If a resolver function returns an error tuple, Absinthe will not resolve that field and adds an error to the response.

This is just one example of how to use Absinthe context. There are many more advanced potential use cases. For example, you might want to automatically set a user’s id as the post author, instead of accepting an author_id in the post_create_input.

Middleware in Absinthe for Elixir

We have used context to authorize a user during resolution. But if we have many fields to handle, this soon becomes too cumbersome. Absinthe provides middleware to handle such cases.

For example, let's assume that several fields require a user to be present when performing an operation. Instead of updating the resolver function for all those fields, wouldn’t it be better if we could just write middleware MyAppWeb.Schema.RequireUser in the field definition?

We can do that using the middleware/2 macro:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  mutation do
    field :create_post, :post_mutation_result do
      arg :post, non_null(:post_create_input)

      middleware MyAppWeb.Schema.RequireUser

      resolve &MyAppWeb.Resolvers.PostResolver.create_post/3
    end
  end
end

The argument to middleware is a module that implements the Absinthe.Middleware behaviour. The module should start a call/2 function that receives an Absinthe.Resolution struct as the first argument.

The second argument is whatever is passed to the middleware/2 macro. We don’t pass anything above, so it's nil by default. The call/2 function should return an updated Absinthe.Resolution struct.

Implement `RequireUser` Middleware

Let’s implement the RequireUser middleware now to stop requests if a user is not present:

defmodule SpendraWeb.Schema.RequireUser do
  @behaviour Absinthe.Middleware

  def call(%Absinthe.Resolution{state: :resolved} = resolution, _config), do: resolution

  def call(%Absinthe.Resolution{context: {user: nil}} = resolution, _config) do
    Absinthe.Resolution.put_result(
      resolution,
      {:error, "You must be logged in to access this API"}
    )
  end

  def call(%Absinthe.Resolution{} = resolution, _config), do: resolution
end

The middle clause is the most important. Here, we receive a context with a nil value for the user. We use Absinthe.Resolution.put_result to mark that the resolution is now complete. It does two things:

Marks the state of the resolution as resolved.
Puts the specified value as the resolution result. Here we use an error tuple to signal that this is an erroneous response.

We also have a special clause at the beginning that checks for the resolution's existing state. If you are using multiple middlewares in your app, this is important. It avoids a middleware running on a resolution that has already been resolved.

Finally, if a resolution's state is not already resolved and we have a user in the context, we just return the original resolution struct without any modifications. This allows execution to proceed.

Chaining Middleware

The great thing about middleware is that it can be chained. For example, we might have several user roles and only want authors to be able to create posts. We can use multiple middleware clauses that each perform a single task before the final resolution.

field :create_post, :post_mutation_result do
  middleware MyAppWeb.Schema.RequireUser

  middleware MyAppWeb.Schema.RequireAuthor

  resolve &MyAppWeb.Resolvers.PostResolver.create_post/3
end

In fact, resolve itself is just a middleware which roughly translates to middleware Absinthe.Resolution, unquote(function_ast).

Another important point about middlewares is that they are executed in the order they are defined in a field. So in the above example, first RequireUser will be executed, followed by RequireAuthor and, finally, the resolver.

It is also possible to use middleware after a resolve call — for example, to handle resolution errors.

Wrap Up

In this post, we created GraphQL mutations with Absinthe. We also added a custom plug to put shared information in an Absinthe context and defined middleware to handle common tasks like authorization.

In the next and final part of this series, we will discuss advanced GraphQL use cases like subscriptions (including how to implement and deliver subscriptions over a WebSocket connection).

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Create and Open a Modal in Phoenix 1.7

Roy Tomeij — 2023-06-20T00:00:00+00:00

Phoenix 1.7 came out this year with a whole host of exciting features, including verified routes and some great built-in Tailwind components. These components are a fantastic start, but they are not made to be a fully general design system. We should expect to modify components to fit our specific needs. However, knowing where to start can be difficult.

In this three-part series, we'll take a fresh Phoenix app and create a working UI using generated components.

In this part, we will add a modal to a page and open it on demand.

Let's get started!

The UI of Our Phoenix App — Setup

The UI we will implement is a list of data and two modals for that data: a create modal and an edit modal. Our example will be a spectacular pet shop called Petacular.

To begin, let's bootstrap the app. If you wish to write the command yourself, first ensure you have the latest Phoenix installer with:

mix archive.install hex phx_new

Then run:

mix phx.new petacular

This will bootstrap the project and generate the core components we'll use in our examples. To illustrate the various steps in this article, I have included a companion repo with commits for each step. This commit shows us everything that gets generated in the above command. Of particular interest is this module which contains all of the generated components we will be exploring.

The components use Tailwind CSS for styling and are passed attributes and/or slots, as appropriate. We will leverage a few below, but first, we will need a database.

Check out this commit for everything we need to get a db working with docker. This commit adds a migration and creates a few schemas we will use for our example: a list of pets and their preferences. Finally, you can see how this commit makes the home page a LiveView page and introduces some very basic styling.

What Is a Modal in Phoenix?

The initial homepage looks like this (found in lib/petacular_web/pages/home_live.ex):

@impl true
def render(assigns) do
  ~H"""
  <h1 class="font-semibold text-3xl mb-4">Pets</h1>
  <PetacularWeb.CoreComponents.button>
    Add New Pet +
  </PetacularWeb.CoreComponents.button>
  """
end

I have included a built-in component from the generated PetacularWeb.CoreComponents module (found in lib/petacular_web/components/core_components.ex) — a button. The button is defined here and it is simple to use:

# in /lib/petacular_web/pages/home_live.ex
...
~H"""
    <PetacularWeb.CoreComponents.button>
      Add New Pet +
    </PetacularWeb.CoreComponents.button>
"""
...

We would love for this to open a modal that contains a form for adding a pet. To do that, let's look at the modal in the PetacularWeb.CoreComponents module.

This is a function component, meaning it does not have a state. It's just a bundle of markup and styling. It has some attrs — for example, attr :show, :boolean, default: false, and one slot.

What Are `attr`s and Slots?

An attr defines data that's expected to pass. Some attrs are required, and some have a default value instead. A slot is space for nested HTML and can be named or not. In essence, slots let you provide your own markup and appear as children of a function component's element.

We can see the slot is called :inner_block, a special name for the modal. Everything inside of the <.modal> tags will be treated as the :inner_block slot. So, in the function docs, the modal component looks like this:

<.modal id="confirm-modal">
  This is a modal.
</.modal>

The text This is a modal. is treated as the :inner_block. In contrast, we can name a slot. Then we designate the contents of that slot by putting content in between two tags that use the slot's name. For example, CoreComponents has a <.header> component with an optional :subtitle slot. To provide a subtitle, we do this:

<CoreComponents.header>
  This is my title.
  <:subtitle>
    This is my subtitle. There are many like it but this one is mine.
  </:subtitle>
</CoreComponents.header>

Making the Modal Appear in Phoenix

The docs for the modal give us an indication of what the modal needs to look for. We need to provide an ID that is targeted by the hide_modal and show_modal functions. Let's look at how they work first. The show modal is defined like this:

# in lib/petacular_web/components/core_components.ex

def show_modal(js \\ %JS{}, id) when is_binary(id) do
  js
  |> JS.show(to: "##{id}")
  |> JS.show(
    to: "##{id}-bg",
    transition: {"transition-all transform ease-out duration-300", "opacity-0", "opacity-100"}
  )
  |> show("##{id}-container")
  |> JS.add_class("overflow-hidden", to: "body")
  |> JS.focus_first(to: "##{id}-content")
end

def show(js \\ %JS{}, selector) do
  JS.show(js,
    to: selector,
    transition:
      {"transition-all transform ease-out duration-300",
       "opacity-0 translate-y-4 sm:translate-y-0 sm:scale-95",
       "opacity-100 translate-y-0 sm:scale-100"}
  )
end

This uses the new(ish) JS module to execute simple JavaScript functions. It accepts and uses an ID to identify the element we want to show. Upon appearing, the show modal does some simple animations to ease it into view and focuses the page onto the first focus-able element, if there is one.

hide_modal does the same, but in reverse:

# in lib/petacular_web/components/core_components.ex

def hide_modal(js \\ %JS{}, id) do
  js
  |> JS.hide(
    to: "##{id}-bg",
    transition: {"transition-all transform ease-in duration-200", "opacity-100", "opacity-0"}
  )
  |> hide("##{id}-container")
  |> JS.hide(to: "##{id}", transition: {"block", "block", "hidden"})
  |> JS.remove_class("overflow-hidden", to: "body")
  |> JS.pop_focus()
end

def hide(js \\ %JS{}, selector) do
  JS.hide(js,
    to: selector,
    time: 200,
    transition:
      {"transition-all transform ease-in duration-200",
       "opacity-100 translate-y-0 sm:scale-100",
       "opacity-0 translate-y-4 sm:translate-y-0 sm:scale-95"}
  )
end

Calling the `show_modal` Function

With a button click, we call the show_modal function to make the modal appear. To close the modal, we need a button on the modal itself that calls hide_modal. Luckily, this is implemented for us, so we don't need to worry about it.

From our function docs, we see that we need some content inside the modal. Like the button we used earlier, the modal uses an :inner_block slot. That means anything inside the modal tags will appear on the page as the :inner_block. We can keep this very simple. Something like the following will work for now:

# in /lib/petacular_web/pages/home_live.ex

<PetacularWeb.CoreComponents.modal id="create_modal">
  <h2>Add a pet.</h2>
</PetacularWeb.CoreComponents.modal>

We can put this inside our homepage and have a click trigger the show_modal function by adding phx-click onto the button we used earlier. Usually, we set phx-click to a string, and clicking it sends an event to the backend using that string as the event name. But we may also provide a JS function or a chain of them:

# in /lib/petacular_web/pages/home_live.ex

<PetacularWeb.CoreComponents.button phx-click={
  PetacularWeb.CoreComponents.show_modal("create_modal")
}>
  Add New Pet +
</PetacularWeb.CoreComponents.button>

Putting it all together, we end up with something like this:

# in /lib/petacular_web/pages/home_live.ex

~H"""
  <h1 class="font-semibold text-3xl mb-4">Pets</h1>

  <PetacularWeb.CoreComponents.modal id="create_modal">
    <h2>Add a pet.</h2>
  </PetacularWeb.CoreComponents.modal>

  <PetacularWeb.CoreComponents.button phx-click={
    PetacularWeb.CoreComponents.show_modal("create_modal")
  }>
    Add New Pet +
  </PetacularWeb.CoreComponents.button>
"""

Now when you click the button, the modal will open! 🎉

See the full diff of changes.

Wrapping Up

In this post, we used Phoenix 1.7's generated core components to create a modal and open it.

In part two, we'll add the edit form.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Absinthe for Large Elixir Applications

Roy Tomeij — 2023-06-06T00:00:00+00:00

In our introduction to Absinthe, we covered the basics of Absinthe and GraphQL for an Elixir application.

Now we'll dig deeper and see how we can customize Absinthe for larger-scale Elixir applications.

Let's get going!

Defining Resolvers for a Large Elixir App

We've seen that creating a query and returning a result in Absinthe is really easy. But if you have big resolution logic, the schema can soon get very heavy. It's common practice to define resolver functions in a separate module for large apps using Absinthe.

First, create a module that defines functions acting as field resolvers:

# lib/my_app_web/schema/resolvers/blog.ex
defmodule MyAppWeb.Schema.Resolvers.Blog do
  def post(%{id: post_id}, _resolution) do
    # complex resolution logic here
    # post = ...
    {:ok, post}
  end
end

Then, update your schema to use this resolver:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  query do
    field :post, :post do
      arg :id, non_null(:id)
      resolve &MyAppWeb.Schema.Resolvers.Blog.post/2
    end
  end
end

This also allows you to unit-test that resolution logic separately from your GraphQL integration tests. We will discuss more about testing schemas later on in this series.

The resolvers additionally help you to reduce code duplication at several places in your schema. For example, let's say you need to expose a field under a different name from what is defined on the struct. We can build the object like this:

object :post do
  # ...
  field :published_at, :datetime do
    resolve fn %Post{} = post, _args, _resolution ->
      {:ok, post.inserted_at}
    end
  end
end

If you find yourself doing that at several places in the schema, it might be better to create a resolver:

defmodule MyAppWeb.Schema.Resolvers.Base do
  @doc """
  Returns a resolver function that returns the value at `field` from `parent`
  """
  def alias(field) do
    fn parent, _args, _resolution -> {:ok, Map.get(parent, field)} end
  end
end

And then use it in your object like this:

object :post do
  # ...
  field :published_at, :datetime, resolve: Resolvers.Base.alias(:inserted_at)
end

Avoiding N+1 Queries in Elixir

Let’s get back to our schema, but with a query that returns a list of posts:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  object :post do
    # ...
    field :author, non_null(:author),
      resolve: &Resolvers.Blog.post_author/3
  end

  query do
    field :posts, non_null(list_of(:post)),
      resolve: &Resolvers.Blog.list_posts/2
  end
end

Here is the full Resolvers.Blog if you are interested.

Now, perform a query that includes the author of each post:

query {
  posts {
    id
    author {
      id
      firstName
    }
  }
}

When this query is executed, we first fetch all posts from the database. Then we fetch the author by their id for each post — that's not very efficient.

But that’s an easy problem to solve. We can just preload the authors in Resolvers.Blog.list_posts/2.

However, what if someone makes a query that doesn’t need an author? We will still be unnecessarily fetching all authors, defeating the whole purpose of having a composable query like this.

One possible solution is to be smart when resolving the initial list of posts and load authors only when the user has selected the author field in the query. If you remember, we get an Absinthe.Resolution struct as the last argument to the resolver function. resolution.definition.selections contains all the selected fields (Absinthe.Blueprint.Document.Field structs).

We can check if the author was selected here, and preload it right there:

defmodule Resolvers.Blog do
  def list_posts(%{}, %Absinthe.Resolution{} = res) do
    posts = Blog.list_posts()

    res.definition.selections
    |> Enum.any?(&(&1.name == "author"))
    |> if do
      {:ok, Repo.preload(posts, :author)}
    else
      {:ok, posts}
    end
  end
end

That works, but if you have many fields (or several nested levels), it can soon become too cumbersome. This is where dataloader can help.

Dataloader to the Rescue!

Dataloader provides an efficient way to load linked data by batching queries. It does this by first performing a full pass through the query resolution phase, collecting all the ids of objects to be loaded. Dataloader then loads them all in one go instead of making a request to the database for each of them separately.

It’s a separate dependency, so first add it to your mix.exs inside deps.

You then need a one-time setup in your schema:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  # ... schema definition here ...

  # Add dataloader to the context
  def context(ctx) do
    loader =
      Dataloader.new
      |> Dataloader.add_source(Blog, Blog.data())
      # ... add other sources here ...

    Map.put(ctx, :loader, loader)
  end

  # Add dataloader to the plugins
  def plugins do
    [Absinthe.Middleware.Dataloader] ++ Absinthe.Plugin.defaults()
  end
end

You can add custom fields to the context through the context method in the schema. The context's value is available to all steps in the GraphQL request cycle (e.g., inside resolvers or middleware) inside the %Absinthe.Resolution{} struct. Context is also where we usually store user details if we authenticate users. Check out the Absinthe Context and Authentication guide to explore this further.

In addition, we add dataloader to plugins using the plugins/0 callback on the schema. This allows dataloader to hook into the resolution pipeline. If you want to learn more about plugins, read the Writing Middleware and Plugins guide for Absinthe.

DataLoader.add_source/3 expects the name of the data source as its second argument and a module implementing the Dataloader.Source protocol. Dataloader supports an Ecto-based data source out of the box, which is what we will need for our example.

Let's update our Phoenix context (Blog in our example) to return the Ecto data source from data/0:

# my_app/blog.ex
defmodule MyApp.Blog do
  def data(), do: Dataloader.Ecto.new(MyApp.Repo, query: &query/2)

  def query(queryable, _params), do: queryable
end

It isn't really documented very well, but if you are feeling adventurous, most of the data-loading magic lies inside the Ecto data source. It uses the query/2 function we passed to generate a base query when it needs to load some data.

For example, if we try to load Author with ids 1 through 5, it will make a single query like from a in Author, where a.id in [1, 2, 3, 4, 5], instead of making 5 different queries.

This function is our opportunity to filter results and return a query that will finally be used to fetch the items. For now, we just return the queryable as it is, which means that we don’t need any special filtering.

Use Dataloader Inside the Absinthe Schema

To use Dataloader inside our schema, we must now modify our object post to use the dataloader helper.

import Absinthe.Resolution.Helpers, only: [dataloader: 2]

object :post do
  # ... other fields
  field :author, non_null(:author),
    resolve: dataloader(MyApp.Blog, :author)
  end
end

The first argument to dataloader/2 is the source name (as registered in the schema).
The next argument is the name of the field in the parent object (author field in parent object post).

Note that the data source should be the one that will resolve the field. So if the post belongs to an author with type MyApp.Accounts.User, you must use dataloader(MyApp.Accounts, :author) as the resolver and support data/0 and query/2 inside the Accounts context.

Here is the full code if you are interested. I know it's a lot to take in, so let's go through the execution of the query above.

Absinthe first invokes our posts resolver (Resolvers.Blog.list_posts/2), and returns the list of posts. Absinthe then checks for the fields it needs inside post and encounters a selection for author. This is where dataloader takes over:

It collects the author_id for all posts that will be returned in our result. Let's say we need to load authors [1, 2, 3, 4, 5].
It then calls MyApp.Blog.query(Author, %{}) to get the initial query. In our example, we are simply returning Author (but in a real application, this could be filtered by business case — for example, if we need only authors that have an active account, we could return where(queryable, [a], a.active), instead of just returning queryable).
Finally, it loads the required ids from the above query — from a in Author, where a.id in [1, 2, 3, 4, 5].

As you can see, we only performed a single query instead of 5 different ones.

Nesting also works out of the box, so if each author has an organization field and we select that in the query, Dataloader will load all organizations in one batch.

Organizing Your Absinthe Schema with Imports

As your schema starts growing, you will soon notice that putting all type definitions and query fields in the same file is not sensible. This is where import_types and import_fields come into play.

The level to split at depends on the size of your API and your application, but it is a common practice to split by business context (the same as your Phoenix contexts).

Here is a structure that works well.

Create a module that contains queries (and another for mutations) related to each model:

# lib/my_app_web/schema/types/blog/post/queries.ex
defmodule MyAppWeb.Schema.Types.Blog.Post.Queries do
  use Absinthe.Schema.Notation

  object :post_queries do
    field :posts, list_of(:post), resolve: Resolvers.Blog.posts/2
    # ... all queries related to post here
  end
end

Create a module for types related to each model. Also import the query and mutation types here.

# lib/my_app_web/schema/types/blog/post.ex
defmodule MyAppWeb.Schema.Types.Blog.Post do
  use Absinthe.Schema.Notation

  import_types(MyAppWeb.Schema.Types.Blog.Post.Queries)
  import_types(MyAppWeb.Schema.Types.Blog.Post.Mutations)

  object :post do
    field :title, not_null(:string)
    # ...
  end

  # all types related to blog here
end

Create a module for types related to each context. This should only import the types from model-specific modules.

# lib/my_app_web/schema/types/blog.ex
defmodule MyAppWeb.Schema.Types.Blog do
  use Absinthe.Schema.Notation

  alias MyAppWeb.Schema.Types

  import_types(Types.Blog.Post)
  # ... import all types related to blog here

  object :blog_queries do
    import_fields(:post_queries)
    # ... import all queries related to blog here
  end

  object :blog_mutations do
    import_fields(:post_mutations)
    # ... import all queries related to blog here
  end
end

Finally, import the context-specific types to your schema.

# lib/my_app_web/schema.ex
defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  import_types(MyAppWeb.Schema.Types.Blog)

  query do
    import_fields :blog_queries
  end

  mutation do
    import_fields :blog_mutations
  end
end

This way, your schema remains very clear, declaring only what it imports. All specific queries are further down the pipeline.

This may seem like overkill for our small API example. But we have been using it in production for a large app with several contexts, and it’s been a boon to keep our schema manageable.

Wrap Up

In this post, the second part of our series on Absinthe, we customized Absinthe for an Elixir application pushing a lot of data. We started by defining resolvers for a big Elixir application and covered how to avoid N+1 queries.

Finally, we dived into Dataloader (which helps to load linked data) in some detail and explored how to organize our Absinthe schema.

Next up, we'll look at creating mutations and subscriptions with Absinthe.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Tackling Performance Issues in Ecto Applications

Roy Tomeij — 2023-05-23T00:00:00+00:00

Ecto can be considered the standard database wrapper and query generator for Elixir, enabling developers to interact with databases efficiently.

However, inefficient queries or misconfigurations still can (and will) happen. Addressing these issues is crucial for a smooth user experience.

In this article, we'll explore three common performance issues in Ecto applications:

N+1 query problem: You get excessive redundant queries from retrieving related data.
Inefficient query execution: Poorly designed queries will strain a database.
Connection pooling and concurrency issues: Bottlenecks and slow response times are caused by configuration or concurrency problems.

We'll discuss how to detect each issue, as well as their common causes and solutions, so you can optimize your Elixir applications for peak performance.

Let's start!

The N+1 Query Problem

The N+1 query problem occurs when an application loads a parent record and its associated child records in separate queries.

This leads to one query for the parent record (1) and one query for each child record (N), resulting in N+1 queries in total. This issue can cause a significant performance hit due to an excessive number of redundant queries.

Detecting the Problem in Elixir

Suppose we have a User schema on a many-to-many relationship with a Role schema. We'll call the schema that associates the two RoleAssignment:

# my_app/lib/my_app/user.ex
defmodule MyApp.User do
  use Ecto.Schema

  schema "users" do
    field :name, :string

    has_many :role_assignments, MyApp.RoleAssignment, foreign_key: :user_id

    timestamps()
  end
end

# my_app/lib/my_app/role.ex
defmodule MyApp.Role do
  use Ecto.Schema
  import Ecto.Changeset

  schema "roles" do
    field :name, :string

    has_many :role_assignments, MyApp.RoleAssignment, foreign_key: :role_id

    timestamps()
  end
end

# my_app/lib/my_app/role_assignment.ex
defmodule MyApp.RoleAssignment do
  use Ecto.Schema

  schema "role_assignments" do
    belongs_to :role, MyApp.Role, foreign_key: :role_id
    belongs_to :user, MyApp.User, foreign_key: :user_id
    timestamps()
  end
end

Imagine that we want to show all the users and their roles on an admin page:

users = Repo.all(User)

users_with_roles =
  Enum.map(users, fn user ->
    role_assignments = Repo.all(from ra in MyApp.RoleAssignment, where: ra.user_id == ^user.id)
    roles = Enum.map(role_assignments, fn ra -> Repo.get!(MyApp.Role, ra.role_id) end)

    %{user | roles: roles}
  end)

As you can see here, we generate N queries to load the associated roles for every user.

To detect the N+1 query problem, you can use tools like Telemetry. Telemetry monitors query counts and identifies queries that are executed multiple times with slight variations.

For example, you wrap the query into a span and then attach a handler to the events to detect it:

# my_app/lib/user_context.ex
defmodule MyApp.UserContext do
  alias MyApp.User
  alias MyApp.Role
  alias MyApp.RoleAssignment

  alias MyApp.Repo

  def get_all_users do
    event = [:my_app, :query]                    # the name of the event
    start_metadata = %{context: "get_all_users"} # metadata to be sent on start

    :telemetry.span(event, start_metadata, fn ->
      users = Repo.all(User)
      result = Enum.map(users, fn user ->
        %{user | roles: get_roles(user)}
      end)

      stop_metadata = %{}                        # metadata to be sent on stop

      {result, stop_metadata}
    end)
  end

  def get_roles_for_user(user) do
    event = [:my_app, :query]                    # the name of the event
    start_metadata = %{                          # metadata to be sent on start
      context: "get_roles_for_user",
      user_id: user.id
    }

    :telemetry.span(event, start_metadata, fn ->
      role_assignments = Repo.all(from ra in MyApp.RoleAssignment, where: ra.user_id == ^user.id)
      roles = Enum.map(role_assignments, fn ra -> Repo.get!(MyApp.Role, ra.role_id) end)

      stop_metadata = %{}                        # metadata to be sent on stop

      {roles, stop_metadata}
    end)
  end
end

We receive two events for every function call: one when the span starts and the other when the span has finished.

With the spans in place, we can now attach a handler to listen for any call to them:

defmodule MyApp.Telemetry do
  # ...
  def handle_user_context_spans([:my_app, :user_repo, start_or_stop], _measurements, _metadata, _config) do

    case start_or_stop do
      :start ->
        # handle span start

      :stop ->
        # Record that a query was executed within the context
    end
  end
  #...
end

New Phoenix applications ship with a telemetry supervisor under <app_web>/telemetry.ex. We can add the handler to its init/1 function:

# my_app/lib/my_app_web/telemetry.ex
defmodule MyAppWeb.Telemetry do
  # ...
  def init(_args) do
    children = [
      # ...
    ]

    :telemetry.attach_many("user_context_handler",
      [
        [:my_app, :user_repo, :start],
        [:my_app, :user_repo, :stop]
      ],
      &MyApp.Telemetry.handle_user_context_spans/4, [])

    Supervisor.init(children, strategy: :one_for_one)
  end
  # ...
end

Here are a couple of strategies to detect N+1:

Count the total number of queries within a span context to analyze anomalies or spikes.
Create a metric, publish an event from the span handler, and, from there, plot data into a dashboard.

It is not an easy problem to spot, but when your application traffic starts to increase, you'll see some symptoms. Here are some additional things you can look for that may indicate you're experiencing an N+1 problem:

High CPU usage on your database systems
All the pages of your web app are fast except for a few
When you open them, lots of the same SQL is being executed

In general, the more instrumentation you have in place, the easier it is to detect N+1 queries.

For more information on N+1 queries and how to use AppSignal to detect them, read our post Performance and N+1 Queries: Explained, Spotted, and Solved.

Using Ecto's Preload

Ecto provides the preload/3 function to load associated records in a single query, avoiding the N+1 query problem.

Here's an example of how to use preload/3. First, add the proper relationship to the model:

# my_app/lib/my_app/user.ex
defmodule MyApp.User do
  # ...

  schema "users" do
    # ...
    many_to_many :roles, MyApp.Role, join_through: MyApp.RoleAssignment
    # ...
  end
end

Now, to fetch all users along with their roles, you can use preload/3 like this:

import Ecto.Query

users = MyApp.Repo.all(
  from(u in MyApp.User,
    preload: [:roles]
  )
)

This will load the user in one query, and roles in another query, regardless of the number of posts a user might have. Great — we go from N+1 to 2!

As an example, an N+1 query with 200 users and 200 roles takes 20 seconds to load on my computer. Using the preload, this number is reduced to 600 milliseconds!

But there is room for even more optimization.

Preloading with Joins

Preloading with joins has the advantage of generating only one query to load all associated records from a database.

In addition, you can also filter or sort the associated records! Just use join/5 along with preload/3.

Following the same example from the last section:

import Ecto.Query

query = from u in User,
  left_join: ra in assoc(u, :role_assignments),
  left_join: r in assoc(ra, :role),
  preload: [role_assignments: ra, roles: r],
  select: u

Repo.all(query)

This fetches users and their assigned roles using a single query, avoiding the N+1 query problem. On my computer, this takes about 100 milliseconds to run.

By leveraging Ecto's preload/3 function and combining it with join/5 when necessary, you can efficiently load associated records and eliminate the N+1 query problem in your Ecto-based applications.

Inefficient Queries in Ecto

Inefficient query execution can result in slow database performance, as poorly designed queries may place unnecessary strain on the database. Now we'll learn how to detect and fix them.

Example 1: Missing Indexes

One of the most common causes of inefficient queries is a lack of indexes.

Suppose we have a Post schema like this one:

# my_app/lib/post.ex
defmodule MyApp.Role do
  use Ecto.Schema
  import Ecto.Changeset

  schema "posts" do
    field :title, :string
    field :body, :string

    field :published_at, Date

    timestamps()
  end
end

Let's fetch all published posts ordered by their publication date:

import Ecto.Query

published_posts = MyApp.Repo.all(
  from(p in MyApp.Post,
    where: is_nil(p.published_at) == false,
    order_by: [desc: p.published_at]
  )
)

If there is no index on the published_at field, this query may become slow when the number of posts in the database grows. The database system will perform a full scan of the table for every query.

To fix this issue, you can add an index to the relevant column using a migration:

defmodule MyApp.Repo.Migrations.AddIndexOnPublishedAt do
  use Ecto.Migration

  def change do
    create index(:posts, [:published_at])
  end
end

There's really no easy way to detect this from the application — it will only start to be noticeable with queries on large tables. From the point of view of an application, it's only a slow query.

In the next section, we'll learn how to monitor the query execution time with telemetry to detect such problems.

Detecting Bad Queries with PostgreSQL and Telemetry

You can use tools like EXPLAIN ANALYZE in PostgreSQL (or similar features in other databases) to detect inefficient queries by analyzing the execution plan and identifying possible bottlenecks.

Additionally, you can use telemetry to monitor query execution times and set up alerts if a query takes too long to execute.

For example, you might add a handler to alert you if a query takes longer than 5 seconds:

# my_app/lib/my_app/telemetry.ex
defmodule MyApp.Telemetry do
  # ...
  def handle_event([:my_app, :repo, :query], measurements, metadata, _config) do
    query_time_ms = measurements[:query_time] / (1_000 * 1_000)
    if query_time_ms > 5_000 do
      # Send an alert or log a warning
    end
  end
  # ...
end

Now we can attach the handler during the telemetry supervisor startup, just like we did for the span handlers:

# my_app/lib/my_app_web/telemetry.ex
defmodule MyAppWeb.Telemetry do
  # ...
  def init(_args) do
    # ...

    :telemetry.attach("query-time-handler", [:my_app, :repo, :query], &MyApp.Telemetry.handle_event/4, [])

    Supervisor.init(children, strategy: :one_for_one)
  end
  # ...
end

Query timeouts can also serve as a bad smell, indicating that inefficient query execution is occurring.

Ecto Connection Pooling and Concurrency Issues

Connection pooling problems can occur when an application tries to open more connections to a database than the pool allows. This can lead to a bottleneck, as processes are left waiting in a queue for an available connection.

Once again, telemetry can help us identify this problem. Ecto already ships with Telemetry support and emits several events that we can listen to and react to. The documentation has data and metadata that Ecto emits for all queries, including queue_time.

We can modify the previous handler example and also monitor long query waiting times in the connection pool queue:

# my_app/lib/my_app/telemetry.ex

  # ...
  def handle_event([:my_app, :repo, :query], measurements, metadata, _config) do
    queue_time_ms = measurements[:queue_time]

    if queue_time_ms > 5_000 do
      # The query waited more than 5s for a connection to be available
    end
  end

Of course, you can always increase the number of connections in an application's configuration:

# config/runtime.exs or config/dev.exs
config :hatch, MyApp.Repo,
  # ...
  pool_size: String.to_integer(System.get_env("POOL_SIZE", "10"))

With the config above, you can control the pool size using the POOL_SIZE env var or the default 10 connections size.

PostgreSQL Deadlocks

Locks are a fundamental mechanism that databases use to ensure data integrity. Still, they can struggle when there's a high volume of updates from different sources in the same timeframe.

A deadlock occurs when two or more processes wait for each other to release a resource, causing all processes to be stuck indefinitely.

How Deadlocks Occur

Deadlocks can happen when two or more processes try to acquire locks on multiple resources in an inconsistent order.

Consider the following schemas:

# my_app/lib/my_app/post.ex
defmodule MyApp.Post do
  use Ecto.Schema

  schema "posts" do
    field :reaction_count, :integer, default: 0
    has_many :reactions, MyApp.Reaction
  end
end

# my_app/lib/my_app/reaction.ex
defmodule MyApp.Reaction do
  use Ecto.Schema

  schema "reactions" do
    field :type, :string
    belongs_to :post, MyApp.Post
  end
end

Now imagine the following scenario using post and reaction schemas:

Process A acquires a lock on Post X.
Process B acquires a lock on Post Y.
Process A tries to acquire a lock on Post Y (but is blocked because Process B holds the lock).
Process B tries to acquire a lock on Post X (but is blocked because Process A holds the lock).

In this case, both processes wait for each other to release the locks, resulting in a deadlock.

If two processes try to add reactions to different posts and update reaction counts at the same time, we could potentially run into a deadlock:

# Process A
task_a = Task.async(fn ->
  Repo.transaction(fn ->
    post_x = Repo.lock!(from(p in Post, where: p.id == ^post_x_id))
    Repo.insert!(%Reaction{type: "like", post_id: post_x.id})
    Repo.update!(Post.changeset(post_x, %{reaction_count: post_x.reaction_count + 1}))
  end)
end)

# Process B
task_b = Task.async(fn ->
  Repo.transaction(fn ->
    post_y = Repo.lock!(from(p in Post, where: p.id == ^post_y_id))
    Repo.insert!(%Reaction{type: "like", post_id: post_y.id})
    Repo.update!(Post.changeset(post_y, %{reaction_count: post_y.reaction_count + 1}))
  end)
end)

Task.await(task_a)
Task.await(task_b)

Most database systems provide mechanisms to detect and automatically resolve deadlocks by rolling back one of the transactions involved.

To prevent deadlocks, you can:

Acquire locks in a consistent order across all processes. For example, you can enforce ordering by post ID:

Repo.transaction(fn ->
  post = Repo.lock!(from(p in Post, where: p.id == ^post_id, order_by: :id))
  Repo.insert!(%Reaction{type: "like", post_id: post.id})
  Repo.update!(Post.changeset(post, %{reaction_count: post.reaction_count + 1}))
end)

Use timeouts when acquiring locks to prevent indefinite waits. In Ecto, you can use the :timeout option with the Repo.transaction/2 function:

Repo.transaction(fn ->
  post = Repo.lock!(from(p in Post, where: p.id == ^post_id))
  Repo.insert!(%Reaction{type: "like", post_id: post.id})
  Repo.update!(Post.changeset(post, %{reaction_count: post.reaction_count + 1}))
end, timeout: 5_000)

By setting a timeout, a transaction will be rolled back if it cannot acquire the necessary locks within the specified time (preventing deadlocks from causing an application to hang indefinitely).

Final Thoughts

From the application point of view, there is only so much that we can actually monitor, since only a handful of entities can evaluate application context like query execution time, queue time, memory consumption, etc.

The best way to know what is happening to your whole stack is to instrument your application with tools like Telemetry or OpenTelemetry, then connect it to your favorite monitoring tool, like AppSignal!

Wrap Up

In this article, we explored common performance issues in Elixir applications using Ecto and provided insights on detecting and addressing these problems.

We started with the N+1 query problem, learning to identify and solve it using Ecto's preload functionality. Then we investigated inefficient query execution, discussing how to optimize queries and use Telemetry for monitoring.

Lastly, we covered connection pooling and concurrency issues, emphasizing the importance of proper configuration, monitoring, and techniques for avoiding deadlocks.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Absinthe

Roy Tomeij — 2023-05-16T00:00:00+00:00

Absinthe is a toolkit for building a GraphQL API with Elixir. It has a declarative syntax that fits really well with Elixir’s idiomatic style.

In today’s post — the first of a series on Absinthe — we will explore how you can use Absinthe to create a GraphQL API.

But before we jump into Absinthe, let’s take a brief look at GraphQL.

GraphQL

GraphQL is a query language that allows declarative data fetching. A client can ask for exactly what they want, and only that data is returned.

Instead of having multiple endpoints like a REST API, a GraphQL API usually provides a single endpoint that can perform different operations based on the request body.

GraphQL Schema

Schema forms the core of a GraphQL API. In GraphQL, everything is strongly typed, and the schema contains information about the API's capabilities.

Let's take an example of a blog application. The schema can contain a Post type like this:

type Post {
  id: ID!
  title: String!
  body: String!
  author: Author!
  comments: [Comment]
}

The above type specifies that a post will have an id, title, body, author (all non-null because of ! in the type), and an optional (nullable) list of comments.

Check out Schema to learn about advanced concepts like input, Enum, and Interface in the type system.

GraphQL Query and Mutation

A type system is at the heart of the GraphQL schema. GraphQL has two special types:

A query type that serves as an entry point for all read operations on the API.
A mutation type that exposes an API to mutate data on the server.

Each schema, therefore, has something like this:

schema {
  query: Query
  mutation: Mutation
}

Then the Query and Mutation types provide the real API on the schema:

type Query {
  post(id: ID!): Post
}

type Mutation {
  createPost(post: PostInput!): CreatePostResult!
}

We will get back to these types when we start creating our schema with Absinthe.

GraphQL API

Clients can read the schema to know exactly what an API provides. To perform queries (or mutations) on the API, you send a document describing the operation to be performed. The server handles the rest and returns a result. Let’s check out an example:

query {
  post(id: 1) {
    id
    title
    author {
      id
      firstName
      lastName
    }
  }
}

The response contains exactly what we've asked for:

{
  "data": {
    "post": {
      "id": 1,
      "title": "An Introduction to Absinthe",
      "author": {
        "id": 1,
        "firstName": "Sapan",
        "lastName": "Diwakar"
      }
    }
  }
}

This allows for a more efficient data exchange compared to a REST API. It's especially useful for rarely used complex fields in a result that takes time to compute.

In a REST API, such cases are usually handled by providing different endpoints for fetching that field or having special attributes like include=complex_field in the query param. On the other hand, a GraphQL API can offer native support by delaying the computation of that field unless it is explicitly asked for in the query.

Setting Up Your Elixir App with GraphQL and Absinthe

Let’s now turn to Absinthe and start building our API. The installation is simple:

Add Absinthe, Absinthe.Plug, and a JSON codec (like Jason) into your mix.exs:

def deps do
  [
    # ...
    {:absinthe, "~> 1.7"},
    {:absinthe_plug, "~> 1.5"},
    {:jason, "~> 1.0"}
  ]
end

Add an entry in your router to forward requests to a specific path (e.g., /api) to Absinthe.Plug:
```
defmodule MyAppWeb.Router do
  use Phoenix.Router

  # ...

  forward "/api", Absinthe.Plug, schema: MyAppWeb.Schema
end
```
The Absinthe.Plug will now handle all incoming requests to the /api endpoint and forward them to MyAppWeb.Schema (we will see how to write the schema below).

The installation steps might vary for different apps, so follow the official Absinthe installation guide if you need more help.

Define the Absinthe Schema and Query

Notice that we've passed MyAppWeb.Schema as the schema to Absinthe.Plug. This is the entry point of our GraphQL API. To build it, we will use Absinthe.Schema behaviour which provides macros for writing schema. Let’s build the schema to support fetching a post by its id.

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  query do
    field :post, :post do
      arg :id, non_null(:id)
      resolve fn %{id: post_id}, _ ->
        {:ok, MyApp.Blog.get_post!(post_id)}
      end
    end
  end
end

There are a lot of things happening in the small snippet above. Let’s break it down:

We first define a query block inside our schema. This defines the special query type that we discussed in the GraphQL section.
That query type has only one field, named post. This is the first argument to the field macro.
The return type of the post field is post — this is the second argument to the macro. We will get back to that later on.
This field also has an argument named id, defined using the arg macro. The type of that argument is non_null(:id), which is the Absinthe way of saying ID! — a required value of type ID.
Finally, the resolve macro defines how that field is resolved. It accepts a 2-arity or 3-arity function that receives the parent entity (not passed for the 2-arity function), arguments map, and an Absinthe.Resolution struct. The function's return value should be {:ok, value} or {:error, reason}.

Define the Type In Absinthe

In Absinthe, object refers to any type that has sub-fields. In the above query, we saw the type post. To create that type, we will use the object macro.

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  @desc "A post"
  object :post do
    field :id, non_null(:id)
    field :title, non_null(:string)
    field :author, non_null(:author)
    field :comments, list_of(:comment)
  end

  # ...
end

The first argument to the object macro is the identifier of the type. This must be unique across the whole schema. Each object can have many fields. Each field can use the full power of the field macro that we saved above when defining the query. So we can define nested fields that accept arguments and return other objects.

As we discussed earlier, the query itself is an object, just a special one that serves as an entry point to the API.

Using Scalar Types

In addition to objects, you can also get scalar types. A scalar is a special type with no sub-fields and serializes to native values in the result (e.g., to a string). A good example of a scalar is Elixir’s DateTime.

To support a DateTime that we'll use in the schema, we need to use the scalar macro. This tells Absinthe how to serialize and parse a DateTime.

Here is an example from the Absinthe docs:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  scalar :isoz_datetime, description: "UTC only ISO8601 date time" do
    parse &Timex.parse(&1, "{ISO:Extended:Z}")
    serialize &Timex.format!(&1, "{ISO:Extended:Z}")
  end

  # ...
end

We can then use this scalar anywhere in our schema by using :isoz_datetime as the type:

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  @desc "A post"
  object :post do
    # ...
    field :created_at, non_null(:isoz_datetime)
  end

  # ...
end

Absinthe already provides several built-in scalars — boolean, float, id, integer, and string — as well as some custom scalars: datetime, naive_datetime, date, time, and decimal.

Type Modifiers and More

We can also modify each type to mark some additional constraints or properties. For example, to mark a type as non-null, we use the non_null/1 macro. To define a list of a specific type, we can use list_of/1.

Advanced types like union and interface are also supported.

Wrap Up

In this post, we covered the basics of GraphQL and Absinthe for an Elixir application. We discussed the use of GraphQL and Absinthe schema, and touched on types in Absinthe.

In the next part of this series, we'll see how we can apply Absinthe and GraphQL to large Elixir applications.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Test Data Libraries for Elixir

Roy Tomeij — 2023-04-25T00:00:00+00:00

In part one of this series, we introduced Elixir test factories and fixtures. Then in part two, we explored using data generation functions.

Now we'll look at some of the best Elixir libraries to use for your test data.

But before we do, let's quickly discuss why test data libraries can be helpful.

Why Choose a Test Data Library for Your Elixir App?

Elixir's built-in language features are more than sufficient for writing simple helpers to build your application test data.

For example, refer to the Ecto Guide to write a factory method pattern API like the one provided by ExMachina.

However, existing test data libraries are convenient as you don't have to write them yourself.

Elixir Libraries for Your Test Data

Here's a quick overview of existing test data libraries and why you might want to use them. We'll look at ExMachina, ExZample, Faker, and StreamData.

Let's start with the most famous one: ExMachina.

ExMachina

The factory library ExMachina created by Thoughbot uses function names and generates an atom used to call factories. For example:

defmodule MyApp.Factory do
  use ExMachina

  def github_repo_factory do
    repo_name = sequence(:github_repo_name, &"repo-#{1}")

    %GitHub.Repo{
      id: 1296269,
      name: repo_name,
      full_name: "octocat/#{repo_name}",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/#{repo_name}",
      url: "https://api.github.com/repos/octocat/#{repo_name}"
    }
  end
end

You can invoke that factory using the functions provided by ExMachina. For example, here's a function that generates a list of resources:

MyApp.Factory.build_list(3, :github_repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-2"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-3"
    # ...
  }
]

The github_repo_factory/0 can be called using the :github_repo atom, which can call any utility function provided by ExMachina and injected by the use ExMachina macro.

One of the most powerful features of ExMachina is the sequence function, which guarantees you'll get the next increment of a sequence each time you call it, and the number will always be unique. It simplifies the caller's job by eliminating the need to explicitly pass a unique name. Using the sequence function automatically avoids unique database constraint issues when calling the same factory multiple times.

ExMachina also lets you add more building strategies, such as a JSON strategy that transforms your data example into JSON. The documentation suggests that you create smaller modules (with macros defining your factories) to break down a large factory module and still serve everything under the same module API. However, I wouldn't necessarily recommend this approach: it's harder to track the factory code definition if it breaks. I would rather have multiple factory modules when using ExMachina.

ExMachina is a well-established, actively maintained library that's widely used in the Elixir community. If invoking factories using the atom naming mechanism fits your preference, it's a solid choice!

Now let's take a look at another tool — ExZample.

ExZample

ExZample is a factory library I wrote which allows you to use defined examples in your struct modules. My goal was to enable developers to organize their factories in any way they want while still having access to convenient functions. For example, if you don't define an example function in GitHub.Repo, you can still build the struct with its default values:

ExZample.build_list(3, GitHub.Repo)
[
  %GitHub.Repo{
    name: nil
    # ...
  },
  %GitHub.Repo{
    name: nil
    # ...
  }
    %GitHub.Repo{
    name: nil
    # ...
  }
]

You can define an example function in the struct module, and ExZample will automatically use it if it's available.

# github/repo.ex
  def example do
    repo_name = sequence(:github_repo_name)

    %GitHub.Repo{
      id: 1296269,
      name: repo_name,
      full_name: "octocat/#{repo_name}",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/#{repo_name}",
      url: "https://api.github.com/repos/octocat/#{repo_name}"
    }
  end

# test/support/test_helper.exs
# Sequences are created on app start
ExZample.create_sequence(:github_repo_name, &"repo-#{1}")

# then you can invoke the data example with ExZample module:
ExZample.build_list(3, GitHub.Repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
]

If you prefer the atom naming mechanism of ExMachina, ExZample.DSL has you covered. You can define your factory in a module like this:

defmodule MyApp.Factory do
  use ExZample.DSL

  factory :github_repo do
    example do
      repo_name = sequence(:github_repo_name)
      %GitHub.Repo{
        id: 1296269,
        name: repo_name,
        full_name: "octocat/#{repo_name}",
        description: "This your first repo!",
        owner_id: 1,
        owner_url: "https://api.github.com/users/octocat",
        private: false,
        html_url: "https://github.com/octocat/#{repo_name}",
        url: "https://api.github.com/repos/octocat/#{repo_name}"
      }
    end
  end

  def_sequence :github_repo_name, return: &"repo-#{1}"
end

# Then use the aliased factory in your tests:
ExZample.build_list(3, :github_repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
]

Here, we use macros provided by ExZample.DSL to define factories using the factory directive. We need to explicitly define the atom name :github_repo. Inside the factory body, we define the example block that builds a struct example.

Although the ExZample library may not be as well-known or extensively tested as ExMachina, ExZample was designed to pick up the exact features you need and ignore the rest. You may want to consider giving it a try. If you have any feedback or suggestions for improvements, feel free to contribute.

Next up, let's see what Faker has to offer.

Faker

Faker generates sample data that looks realistic but is fake. So, instead of using a sequence like those provided by ExMachina or ExZample, we can use Faker to generate false but genuine-looking data:

# test/support/test_helper.exs
# Ensure you start Faker before running your tests
Faker.start()

Faker.Internet.slug()
"sit_et"

Faker.Internet.slug()
"deleniti-consequatur"

Faker.Internet.slug()
"foo-bar"

The random nature of data generation can sometimes result in duplicate names, which may cause flaky tests. However, this is likely to occur only in rare situations, so it's still worth relying on Faker. Faker has an impressive collection of data samples and generation algorithms, including IP addresses, e-mails, URLs, addresses, and even Pokémon names. It's a great library to use in combination with the others presented here.

Finally, let's take a quick look at StreamData.

StreamData

StreamData generates a bunch of data samples for you. Unlike Faker, StreamData doesn't aim to generate realistic values. It creates raw, random data. You ask for a raw data type, and you get it. For example:

Enum.take(StreamData.string(:alphanumeric), 3)
#=> ["AcT", "9Ac", "TxY"]

StreamData returns a data generator that produces an infinite amount of random data. It implements the Enumerable protocol, so you can filter data using the Enum functions:

StreamData.string(:alphanumeric)
|> Enum.filter(&(String.length(&1) >= 5 and String.length(&1) <= 10))
|> Enum.take(1)
["hygT78ch"]

In the example above, we're only interested in grabbing strings between 5 and 10 characters long. StreamData was designed to be used with property testing, but you can also use its powerful data samples to build data examples for regular tests.

General Thoughts and Suggestions

I often wonder if it's worth deviating too far from the Elixir standard library when using a certain pattern. So I would like to invite you to aim for the following in your next Elixir project:

Try using your application's functions and rules to build test data. Create wrappers to simplify their usage if needed, with convenient defaults for testing.
When your application doesn't control the accuracy of your data, try to create example functions on the struct modules and use them to test your code.
Use Faker or StreamData to add some randomness, enriching your test data and making your tests less dependent on specific data.

If these three points still aren't enough for your data testing needs, embrace a factory library to bring more convenience and structure to your test suite.

If you try the lean suggested approaches, you might realize how little you really need to be able to generate test data and have a healthy test suite.

Wrapping Up

In part one of this series, we summarised the ins and outs of Elixir test factories and fixtures. In the second part, we focused on generating data functions.

Finally, in this third and last part, we explored some test data libraries you can use for your Elixir application, including ExMachina, ExZample, Faker, and StreamData.

I hope you found this series helpful.

Good luck with your testing!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Mocking Tools for Elixir

Roy Tomeij — 2023-04-11T00:00:00+00:00

A well-written test suite is a big part of any successful application. But let's say you rely on an external dependency for some parts of your app (for example, an external API for fetching user information). It then becomes important to mock that dependency in the test suite to prevent external API calls during testing or to test specific behavior.

Several frameworks help reduce the boilerplate and make mocking safe for Elixir tests. We will explore some of the major mocking tools available in this post.

Let's get started!

Test Without Mocking Tools in Elixir

Depending on the scope of your tests, you might not need to use any external mocking tools. You can use test stubs or roll your own server instead.

Test Stubs

The simplest way to stub/mock out some parts from a function call is to pass around modules that do the actual work and use a different implementation during the tests.

Let’s say you need to access the GitHub API to fetch a user’s profile with an implementation like this:

defmodule GithubAPI do
  def fetch_user(username) do
    case HTTPoison.get("https://api.github.com/users/#{username}") do
      {:ok, response} ->
        parse(response)
      {:error, reason} ->
        {:error, reason}
    end
  end
end

We know the implementation relies on HTTPoison to make actual calls to the GitHub API. To mock it during tests, we can update the implementation to pass around an http_client and use a different one during tests.

Something like this:

defmodule GithubAPI do
  def fetch_user(username, http_client \\ HTTPoison) do
    case http_client.get("https://api.github.com/users/#{username}") do
      #... handle results
    end
  end
end

This approach aligns with the Law of Demeter — a module should only have limited knowledge of the objects (in this case, the HTTP client) it works on. This approach is nice because it decouples the GithubAPI module from the HTTP client implementation.

Our GithubAPI users can continue using it in the same way. But for unit-testing GithubAPI, we can pass in our custom http_client that returns just the results we need. For example:

defmodule GithubAPITest do
  defmodule GithubHTTPClientUser do
    def get("https://api.github.com/users/octocat") do
      {:ok, %HTTPoison.Response{status_code: 200, body: ~s<{"login": "octocat"}>}}
    end
    def get("https://api.github.com/users/unknown") do
      {:error, %HTTPoison.Error{message: "Not Found"}}
    end
  end

  test "can fetch a user" do
    {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat", GithubHTTPClientUser)
  end

  test "handles errors when fetching a user" do
    {:error, %HTTPoison.Error{}} = GithubAPI.fetch_user("unknown", GithubHTTPClientUser)
  end
end

That works, and you have 100% test coverage! 🎉 But as you can already imagine, while this works well for small tests, it will become increasingly difficult to manage if our GithubAPI class grows to include other features.

Another similar strategy is to use application configuration instead of passing around the client modules. This is especially useful when you need to mock out the API from all the tests, even when this API is called from other internal modules.

We can update our code to this:

# github_api.ex
defmodule GithubAPI do
  @http_client Application.compile_env(:my_app, GithubAPI, []) |> Keyword.get(:http_client, HTTPoison)

  def fetch_user(username) do
    case @http_client.get("https://api.github.com/users/#{username}") do
      #... handle results
    end
  end
end

# test/support/mock_github_http_client.ex
defmodule MockGithubHTTPClient do
    # ... All mock implementations here
end

# config/test.exs
config :my_app, GithubAPI, http_client: MockGithubHTTPClient

The good thing is that you don’t need to worry about mocking out GithubAPI calls in each test case by manually passing the HTTP client.

This is especially important when the actual API calls are nested inside other functions. For example, if your application automatically fetches the user from GitHub after a new account is created, you don’t need to mock GithubAPI everywhere you create new users. But it still has the same disadvantages as the previous strategy. Plus, the mocks must be generic enough to be used throughout the test suite.

Roll Your Own Server

This one is interesting. Since plug and cowboy make it really easy to roll out an HTTP server, instead of mocking out the HTTP Client, we can start our own server during tests and respond with stubs instead.

If you want to learn more, check out:

In the strategies discussed above, quite a bit of boilerplate is involved in setting up the tests. And if you need a way to validate that a specific function was called with specific arguments, you need to do additional work.

If you just have a small external dependency that you need to mock out, this might work well for you. But if there are complex edge cases and branches that you need to test out, mocking tools can help simplify the complexity of writing and maintaining those mocks in the long run.

Elixir Mocking Tools

Mock

Mock is the first result you will see when searching “Elixir Mock”, and is a wrapper around Erlang’s meck that provides easy mocking macros for Elixir.

With Mock, you can:

Replace any module at will during tests to change return values.
Pass through to the original function.
Validate calls to the mocked functions.
Check the complete call history, including arguments and results for each call.

It is a very powerful tool, and the macro with_mock makes mocking during tests really easy.

Let’s see how we can rewrite our test case to validate GithubAPI.fetch_user:

defmodule GithubAPITest do
  # This is important, Mock doesn't work with async tests!
  use ExUnit.Case, async: false

  import Mock

  test "can fetch a user" do
    with_mock HTTPoison, [get: fn _url -> {:ok, %{status_code: 200, body: ~s({"login": "octocat"})}} end] do
      {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
      assert_called HTTPotion.get("https://api.github.com/users/octocat")
    end
  end
end

Very sleek. There's no need to define and pass boilerplate modules around or fiddle with the config just for tests. Our implementation is completely isolated from the test code and needs no special changes to fit the tests. And if we need to validate that a function was called with specific arguments, we can do that as well with assert_called.

One of the major drawbacks of this strategy is that you cannot use it with async tests. It doesn’t prevent you from using Mock inside asynchronous tests, so this can lead to hard-to-track flaky tests.

In my opinion, Mock works well for certain types of tests. I usually find myself reaching for it when we need to mock external libraries that we have no control over. For example, let’s say we use an external library to fetch a user’s GitHub profile instead of our custom GithubAPI. Something like this:

def create_user(github_username) do
  Tentacat.Users.find(Tentacat.Client.new(), github_username)
  |> case do
    {:ok, user} ->
      # ... create user
    {:error, error} ->
      # ... handle error
  end
end

Now, if we dig into the library’s documentation/code, we can see that it uses HTTPoison to make the eventual request to the API. But we don’t want to — or have a way to — customize that client just for tests. Here, with_mock can easily help mock out that call so that we don’t request the API.

In fact, a much better approach, in this case, is to use with_mock to simply mock out the Tentacat.Users.find call altogether. That saves you from having to dig into the library's internal code (which can change with any update) and solely relying on mocking out its public interface.

Mox

We saw above how easy it is to mock some methods out and have our tests pass. We sprinkle these calls in a few places to mock HTTPoison, and we are done.

But what if we later decide that HTTPoison isn’t fast enough and want to switch to another HTTP client implementation? As expected, all our tests will fail — we have to go back and fix them. Even worse, what if the API for HTTPoison changes, but since we mocked it out, our tests never failed, and we pushed something that didn’t work to production?

Mox helps get around these issues by ensuring explicit contracts. Read Mocks and Explicit Contracts for more details.

Using our GithubAPI example above, this is how we need to set up the tests with Mox.

Mock the External Client

We first convert our API client into a Behaviour to define the explicit contract the API client should follow.

# lib/my_app/github/api.ex
defmodule MyApp.GithubAPI do
  @callback fetch_user(String.t()) :: {:ok, Github.User.t()} | {:error, Github.Error.t()}

  def fetch_user(username), do: impl().fetch_user(username) do

  defp impl(), do: Application.get_env(:my_app, GithubAPI, GtihubAPI.HTTP)
end

Next, we define the API Client that makes the actual request to fetch the user.

# lib/my_app/github/http.ex
defmodule MyApp.GithubAPI.HTTP do
  @behaviour MyApp.GithubAPI

  @impl true
  def fetch_user(username) do
    case HTTPoison.get("https://api.github.com/users/#{username}") do
      {:ok, response} ->
        # Parse response here...
        {:ok, user}
      {:error, error} ->
        # Handle error here...
        {:error, reason}
    end
  end
end

Finally, in our test helper, we define a mock MyApp.GithubAPI.Mock for the API and set it in our application environment.

# test/test_helper.exs
Mox.defmock(MyApp.GithubAPI.Mock, for: MyApp.GithubAPI)
Application.put_env(:my_app, GithubAPI, MyApp.GithubAPI.Mock)

Now we can refactor the test case to use Mox for mocking out the call to the external API:

# test/my_app/github/api_test.exs
defmodule GithubAPITest do
  use ExUnit.Case, async: true

  import Mox

  setup :verify_on_exit!

  test "can fetch a user" do
    MyApp.GithubAPI.Mock
    |> expect(:fetch_user, fn "octocat" -> {:ok, %Github.User{login: "octocat"}} end)
    assert {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
  end
end

There are several interesting aspects to the above code. Let's break it down.

First, we use the expect function to define an expectation on the API. We expect a single call to fetch_user with the argument "octocat", and we mock it to return a {:ok, %Github.User{}} tuple.

Now, when we call GithubAPI.fetch_user/1 in the test, it will reach for that mocked expectation instead of the default implementation. Thus, we can safely assert the result of the call without making calls to the actual API.

The verify_on_exit! function as setup ensures that all expectations defined with expect are fulfilled when a test case finishes. So if we define an expectation on fetch_user and it isn't actually called (e.g., if the implementation changes later), we see the test fail.

Finally, notice that we don't need to mark the test as non-async here. That’s because Mox supports mocking inside async tests. So you can define two completely different mocks in two test cases, and they will both pass, even if they run simultaneously.

You can also define a global stub to avoid providing mocks in every test. This is useful if your client is being used in several places and you don’t want to explicitly define and validate expectations everywhere.

To do this, just update your ExUnit case template:

defmodule MyApp.Case do
  use ExUnit.CaseTemplate

  setup _context do
    Mox.stub_with(MyApp.GithubAPI.Mock, MyApp.GithubAPI.Stub)
  end
end

And create a stub with some static results:

defmodule MyApp.GithubAPI.Stub do
  @behaviour MyApp.GithubAPI

  @impl true
  def fetch_user("octocat"), do: {:ok, %Github.User{login: "octocat"}}
end

Now every time you use MyApp.Case in a test case, you don't need to manually mock calls to GithubAPI — they will automatically be forwarded to the stubbed module. This works well when you have calls to the GithubAPI that you will hit across several test suites. With a stub, you can be sure that such calls always return a specific and stable response without having to mock them out manually.

Test the External Client

If you are following along closely, you will notice that we didn’t actually test the MyApp.Github.HTTP module at all. Don’t worry, we won’t leave it untested. But the recommended way here is to do integration tests instead of using mocking at that level. We will also configure it so that these tests don’t run when you run the full test suite.

# test/my_app/github/http_test.exs
defmodule MyApp.GithubAPI.HTTPTest do
  use ExUnit.Case, async: true

  # All tests will ping the API
  @moduletag :github_api

  # Write your tests here
end

# test/test_helper.exs
ExUnit.configure exclude: [:github_api]

The next step is to set up your CI pipeline to ensure that these tests run when required. For example, you can configure it to run on all pull requests targeting main to avoid reaching the external API on every commit. You can also check that your CI pipeline runs before anything is merged to the main branch. To do this, use the include command line flag when running mix test.

mix test --include github_api

There are several pros to the above strategy. You are now testing all parts of the application, including the calls to the external API (this part is not exactly dependent on Mox, but since we mocked out a significant part of our HTTP client, we must test it separately). We can also define global stubs, specific mocks, and expectations only when required, which makes most of our test suite very clean and concise.

The major drawback is that it requires quite some setup — you need a new behaviour with @callbacks for each public method you want to mock out. You have to implement that behaviour inside your module and finally set up Mox to mock out that implementation during tests.

And since we are now reaching the external API, the tests can be flaky, depending on the API's availability.

Mimic

If you are used to Mocha for other languages, you can check out Mimic. It lets you define stubs and expectations during tests by keeping track of the stubbed module in an ETS table. It also maintains separate mocks for each process, so you can continue using async tests. It’s a great alternative to Mock — but that also means the same caveat applies: be careful about what you mock.

Here’s how a sample test looks with Mimic:

# test_helper.exs
Mmimc.copy(HTTPoison)
ExUnit.start()

# github_api_test.exs
defmodule GithubAPITest do
  use ExUnit.Case, async: true
  use Mimic

  test "can fetch a user" do
    url = "https://api.github.com/users/octocat"
    expect(HTTPoison, :get, fn ^url -> {:ok, %{status_code: 200, body: ~s({"login": "octocat"})}} end)
    assert {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
  end
end

expect/3 automatically verifies that the function is called. And all expect and stub calls can be chained to make up clear mocking code.

Use Mocking Carefully for Your Elixir App

Mocking is an important and necessary part of any test suite. But the thing to remember about mocking is that it is imperative to do it right.

For example, it might be tempting to mock out an internal API to quickly simulate something for a test. And yes, it works for quick unit tests. But then, someone else (or maybe you) comes along a while later and changes that something that you mocked earlier.

Your tests still pass since they use the mocked value. But in practice, the thing is now broken, and it will be caught much later in the feedback loop (or worse still, be shipped to the user as-is).

Wrap Up

In this post, we explored several mocking strategies you can use in Elixir tests. The safest (and the one you should consider using first) is Mox, as it forces mocked modules to have a defined behavior. Mox can therefore catch issues that arise from API changes during compilation.

Reach out for Mimic or Mock when you need to mock external libraries you don't have any control over.

Until next time, happy mocking!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Debugging Phoenix LiveView with open_browser/2

Roy Tomeij — 2023-03-28T00:00:00+00:00

In this blog post, you'll see how useful open_browser/2 is when debugging LiveView tests. In addition, we'll give a brief introduction to testing LiveView.

Let's get started!

Introducing LiveView Testing

Testing during the software development process is one way to build confidence and ensure your application will work as expected.

The ability to easily write meaningful tests is an important factor for any framework, regardless of programming language.

Developers can easily test the LiveView framework's component, life-cycle, and behavior by writing LiveView tests with pure Elixir, since it uses ExUnit (a built-in testing framework) for all its testing. We can be confident in writing LiveView tests that are fast, concurrent, and stable.

You can test the functionality of your live views' behavior through the help of the Phoenix.LiveViewTest module, which offers convenient functions without the need to introduce JS testing frameworks. The test helpers assist us in writing meaningful tests for our LiveView modules with ease and speed.

Sample Feature in LiveView

Let's write a sample feature to use as our testing example.

We shall add a form to our application that allows users to enter their email address and password when attempting to register. This feature will only focus on the rendered HTML and not include the ability to add users into the system (the backend part of taking user params and adding them into the data store).

Let's begin!

First, start by creating a new LiveView application:

mix phx.new sample_live --live

NB: You can use an existing LiveView application if you have one.

The mix phx.new command with the --live flag will create a new application with LiveView installed and configured.

Add a live path in the router.

# lib/sample_live_web/router.ex

scope "/", SampleLiveWeb do

  live "/user_registration", RegistrationLive

end

Create registration_live.ex inside the sample_live_web folder.

# lib/sample_live_web/registration_live.ex

defmodule SampleLiveWeb.RegistrationLive do

  use SampleLiveWeb, :live_view


  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  def render(assigns) do
    ~H"""

      <.form let={f} for={:changeset} id={"registration-form"} >

        <%= label f, :email %>
        <%= text_input f, :email, id: "email-input" %>
        <%= error_tag f, :email %>
        <%= label f, :password, id: "password-input" %>
        <%= password_input f, :password %>
        <%= error_tag f, :password %>
        <%= submit "Save" %>
      </.form>

    """
  end

end

In our LiveView application module, two callbacks have been defined: mount/3 and render/1.

The mount/3 callback expects three arguments — params, session, and liveview socket — and returns {:ok, socket}. When the LiveView page is rendered, the mount/3 callback will be invoked twice: once to perform the initial page load and again to establish the live connection.

The render/1 callback is responsible for displaying the HTML template/content — in this case, the added registration form. It receives the socket assigns and must return a template passed inside the ~H sigil. Whenever there is a change in our LiveView, the render/1 callback will be invoked.

In our template, we have defined a sample form where a user can enter their email and password.

Let's start the server and open the registration form in our browsers. You should see something similar to this:

Display the Feature on the Browser while Testing

When working on an HTTP-based web application, we usually write integration tests to validate passing expected attributes and properties to parts of our application.

We'll write an integration test validating user interactions with our application. The test should verify that when a user visits the /user_registration page, they can see the registration form properties/attributes rendered. In addition, we'll explore how we can use open_browser/2 to debug our LiveView test.

First, let's write a test for our form implemented above that will verify the HTML rendered. In here, we shall use open_browser/2 to verify the displayed form. The implemented registration form has an HTML id ("registration-form") that should help us select the element in our test.

# test/sample_live_web/registration_live_test.exs

defmodule SampleLiveWeb.RegistrationLiveTest do
  use SamplLiveWeb.ConnCase
  import Phoenix.LiveViewTest

  test "user can see registration form", %{conn: conn} do
    {:ok, view, html} = live(conn, "/user_registration")

    html =
      view
      |> element("#registration-form")
      |> open_browser()
      |> render()

    assert html =~ "Email</label>"
    assert html =~ "Password</label>"

  end

end

import Phoenix.LiveViewTest module gets access to the test helper functions.

In the sample feature, when a user is on the application and wants to visit the registration page, they will navigate to /user_registration. A similar thing happens here — we shall navigate the user to /user_registration using live/2, which performs a regular get(conn, path) and then upgrades the page to LiveView. It takes in the conn and the path then returns a three-element tuple with :ok, liveview process, and the rendered HTML.

open_browser/2 expects a view or element as the first argument. In the test, I have opted to pass an element, and this can be done by invoking element/3, passing the view and the query selector to it. element/3 returns an element which is then passed to open_browser/2. If open_browser/2 finds the form element with the query selector(#registration-form) within the LiveView page, we expect the following:

The default browser to be opened displaying the HTML of the form element.
The form element to be returned. The form element returned can be passed to render/1, which takes in a view_or_element and returns an HTML string of the view that we can finally assert in the test.

Run the test:

mix test test/sampl_live_web/registration_live_test.exs

We should expect the default browser to open and our test to pass.

Now, let's make the user enter their email and password by using form/3, which takes in the view, query selector, and form data, and then returns a form element. But this time we'll use a query selector that doesn't exist to demonstrate how open_browser/2 can be useful in debugging LiveView tests.

# test/sample_live_web/registration_live_test.exs

defmodule SampleLiveWeb.RegistrationLiveTest do
  use SamplLiveWeb.ConnCase
  import Phoenix.LiveViewTest

  test "user can see registration form", %{conn: conn} do
    {:ok, view, html} = live(conn, "/user_registration")

    view
    |> element("#registration-form")
    |> open_browser()

    view
    |> form("#wrong_registration-form", user: %{email: "hello@email.com", password: "hello123"})
    |> render()

  end
end

Let's run the test:

** (ArgumentError) expected selector "#wrong_registration-form" to return a single element, but got none within:

<main class="container"><p class="alert alert-info" role="alert" phx-click="lv:clear-flash" phx-value-key="info"></p><p class="alert alert-danger" role="alert" phx-click="lv:clear-flash" phx-value-key="error"></p><form action="#" method="post" id="registration-form" phx-submit="save"><input name="_csrf_token" type="hidden" value="CHwQRnBHAnF7JDQEXQk3AXx8QQ0hZxceYLY-91nA2PLS3bB13Q0HC8CQ"/><label for="registration-form_email">Email</label><input id="email-input" name="user[email]" type="text"/><label for="registration-form_password">Password</label><input id="password-input" name="user[password]" type="password"/><button type="submit">Save</button></form></main>

We notice that our test is failing with an ArgumentError. In this test, the form/3 function returns a form element with the query selector #wrong_registration-form. This form element is then passed to render/1, which expects the form element with the query selector #wrong_registration-form passed to it as the first argument to return a form element (but it gets none within the LiveView page). The test has made it clear that we are using the wrong query selector.

Our form template is very short, so it's easy to identify our mistake. Even from the HTML string displayed on the terminal after running the test, it's easy to point out the query selector we should have used.

Imagine working with larger HTML templates, though. It will be cumbersome to go through the HTML string to try and identify the mistake. Let's examine how open_browser/2 can come in handy in that case.

First, let's add open_browser/2 after form/3:

# test/sample_live_web/registration_live_test.exs

defmodule SampleLiveWeb.RegistrationLiveTest do
  use SamplLiveWeb.ConnCase
  import Phoenix.LiveViewTest

  test "user can see registration form", %{conn: conn} do
    {:ok, view, html} = live(conn, "/user_registration")

    view
    |> element("#registration-form")
    |> open_browser()

    view
    |> form("#wrong_registration-form", user: %{email: "hello@email.com", password: "hello123"})
    |> open_browser()
    |> render()

  end
end

This way, we expect the browser to open with the displayed registration form — but apparently, that won't be the case when we run our test. Just like the render/1 function, open_browser/2 expects the form element with the query selector #wrong_registration-form passed to it, but it doesn't exist.

The other option is to debug the LiveView page where the registration form is rendered. In the test, we shall call open_browser/2 and pass the view returned after invoking live/2 to it. Earlier on, I mentioned that open_browser/2 could either take in a view or an element. The option to use an element is failing due to the wrong query selector. So using view can be a better option in our case to help us understand why the test is failing.

# test/sample_live_web/registration_live_test.exs

defmodule SampleLiveWeb.RegistrationLiveTest do
  use SamplLiveWeb.ConnCase
  import Phoenix.LiveViewTest

  test "user can see registration form", %{conn: conn} do
    {:ok, view, html} = live(conn, "/user_registration")
    open_browser(view)
  end
end

The debug process will be as follows:

Run the test. This time, we should expect the default browser to be opened and our registration form displayed.
Use the element inspector in the browser to find the correct form selector.

From the image above, we can see that the form element under the inspect elements section highlighted in blue has a different query selector (the id attribute value) specified to the one passed in form/3 in our test. Our test expected the registration_form query selector. If we now refactor our test and pass the required query selector, it will definitely pass.

We have seen that we can use open_browser/2 as a debugging tool for our tests. With open_browser/2, we can easily verify that the correct HTML elements are rendered to users. If the correct view_or_element passes, the browser opens with the displayed HTML.

Monitor Your Phoenix LiveView App in Production with AppSignal

If you also want to keep track of your live view performance in production, you can set up LiveView instrumentation in AppSignal. Once you've integrated AppSignal with Phoenix LiveView, you'll be able to monitor incoming HTTP requests.

Check out our docs for Phoenix LiveView for more information.

We recommend using our automatic LiveView instrumentation in the majority of cases.

Wrapping Up

The introduction of open_browser/2 has made debugging LiveView tests easy and efficient. In this post, we introduced how to use open_browser/2 when testing to verify that the correct HTML is rendered within your live view.

Happy debugging!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Generating Data Functions in Your Elixir App

Roy Tomeij — 2023-03-21T00:00:00+00:00

In the first part of this series, we explored the ins and outs of Elixir test factories and fixtures. However, test factories bypass the rules of your Elixir application. Let's turn our attention to how we can avoid this by using data generation functions.

Let's dive straight in!

Creating Data With Your Elixir App's Public APIs

We'll use application code to generate test data. One of the biggest benefits of this method is that the generated data will be consistent with your application's rules. However, you need to write an extra layer of functions that your tests can easily use.

The Test Pyramid Strategy for Your Elixir App

Before jumping into the code, it's important to think about your application's public APIs and how data will be generated for each test strategy.

Saša Juric wrote about how to keep a maintainable test suite in Elixir. In the post, he described how the most important public API he needed to test in his app was the HTTP API layer, which is how his users interact with the app. For his app's context, it was worth having a lot of test HTTP requests, even though it might have some drawbacks on the test speed performance. In his test pyramid strategy, Saša favored interface-level tests over lower-level ones.

The image above shows different strategies you might choose for your app.

You might want a balanced pyramid with a good amount of tests in all layers or favor unit tests above everything else.

The tests in the top layer are slower and more difficult to maintain, and they overlap with lower-layer tests.

The high-level tests ensure that all modules work together, while the lower-level tests can tell you exactly which module or function is not working properly.

Your application's test pyramid strategy might have more layers with different names — for example, the term "unit" might mean completely different things depending on who you ask. Each strategy has its own trade-offs, and discussing them is out of the scope of this post.

The most important thing to take away, though, is that you need to understand the layers of your Elixir application and the type of data needed to write accurate functions that generate test data.

I'll give you some examples, but ultimately, it's up to you to decide what works best for your team and source code.

Extract Business Logic from Your Web Code

It's a well-known and common practice to split business logic from web code in web applications built with the Phoenix framework. In Phoenix, we usually call these business modules "context" modules.

These modules typically interact with databases, external services, other context modules, and a lot of other functions. The functions in context modules are usually the public API for your web layer.

An Example Using Helpers in Elixir

If you want to invest in a lot of tests on the context level, you can create helpers for the most-needed resources. For example, let's say you have these modules:

# lib/my_app/accounts.ex
defmodule MyApp.Accounts do
  def signup_user(username, password) do
  # ...
  end
end

# lib/my_app/profiles.ex
defmodule MyApp.Profiles do
  def create_author_profile(user) do
  # ...
  end
end

# lib/my_app/profiles.ex
defmodule MyApp.News do
  def create_post(post_contents, author) do
  # ...
  end
end

Here, we have the Accounts, Profiles, and News contexts. Suppose we want to test the MyApp.News.create_post/2 function. We first need to create an Author profile. To create an Author profile, we need a User account.

If we have more functions in the News context, or if more contexts need an Author, we always have the tedious task of creating these chained struct relationships.

We can create helpers and make them available to our tests that need valid Author and User structs in the system. To make these helpers only available in the :test environment, we need to tell Mix, Elixir's project build tool, where to find these files. You can update mix.exs with the following configuration:

elixirc_paths: (
  case Mix.env() do
    :test -> ["lib", "test/support"]
    _ -> ["lib"]
  end
 )

We tell Mix to compile the files in the test/support directory with the lib directory when building the :test environment. Of course, you can use any directory name you want, but test/support is a widely used convention. This way, you can create modules only available for a :test environment. Here's an example of some helpers:

# test/support/helpers.ex
defmodule MyApp.Helpers do
  def create_user(opts \\ []) do
    username = Keyword.get(opts, :username, "test_user")
    password = Keyword.get(opts, :password, "p4ssw0rd")

    {:ok, user} = MyApp.Accounts.signup_user(username, password)
  end

  def create_author(opts \\ []) do
    user = Keyword.get_lazy(opts, :user, &signup_user/0)

    {:ok, author} = MyApp.Profiles.create_author_profile(user)
  end
end

In the MyApp.Helpers module above, we create wrappers over our application's core API with convenient defaults to use in tests. In real life, we wouldn't create users with the password "p4ss0wrd" by default, but for our test scripts, it's fine.

We also use Keyword.get/3 to overwrite important attributes of the created resources. This avoids unnecessary side effects, especially when the caller provides us with a user to the create_author/1 helper. That's why we use Keyword.get_lazy/3 in the create_author/1 helper. get_lazy will only invoke the signup_user function if the :user key doesn't exist.

Invoking and Customizing Helpers in Elixir

We can also be strict with data patterns here since we don't expect the creation of users or authors to fail. The caller can invoke and customize these helpers for the needs of the test. For example:

user = create_user(username: "test_user_2")

create_author(user: user)

When writing your tests, you can use the examples above and write something similar:

# test/news/news_test.exs

# module definition and stuff

alias MyApp.Helpers

# maybe other tests in the middle

test "creates new post with given the contents" do
  author = Helpers.create_author()

  assert {:ok, post} = News.create_post("My first post!", author)
end

In the code above, we use alias MyApp.Helpers to make our helper module functions accessible with a few characters. It makes the use of these functions as convenient as the test factories provided by ExMachina.

A cool advantage of this approach is that if your editor uses a language server like ElixirLS, you can quickly discover or navigate these functions easily. The build(:user) pattern doesn't allow the editors of today to track the definition code directly.

Breaking the Helpers Module Down Into Other Modules

As the Helpers module grows and gets more complex, you might want to break it down into different modules.

For example, you could break it down by context - AccountsHelper, ProfilesHelpers, etc. The best naming and file organization depends on each application's needs, so I'll leave it up to you.

The Helpers example discussed here might have left you thinking: "This looks like factories." And you're not wrong!

This is a different implementation of test factories. Instead of creating examples of data uncoupled with your application's rules, here we tie your application's rules and your test examples together.

It works like the factory pattern because your tests aren't coupled with the way the underlying struct is built.

This example satisfies the demands of the context modules layer, but what about the other layers? In the next section, we'll explore when using public APIs isn't enough and how to generate data examples for these cases.

Beyond Public APIs: Data Examples

One of the main advantages of using your Elixir app's public API to generate data for your tests is that the data will comply with your app's rules and database constraints.

While the speed and coupling with the database may not always be ideal, this is a small price to pay to ensure your data's validity.

As your app grows in complexity, you may need to add a layer of tests that run in memory to improve performance. It is also not uncommon for systems to talk to other systems through network APIs. In these cases, your application likely won't have enough control over the rules to build valid data. You may need to rely on API specifications provided by the remote system and snapshot some examples to use in your tests.

Data examples built in memory and decoupled from database or application rules can be extremely helpful.

Let's explore a lightweight — and somewhat controversial — method of generating data examples within your application modules.

Writing Data Examples in Data Definition Modules

Years ago, a colleague invited me to watch an episode of Ruby Tapas presented by Avid Grimm. Inspired by the book Practical Object-Oriented Design by Sandi Metz, Grimm talks about writing data examples in the modules where the data definition lives. This can serve as executable documentation and also be used in tests.

Some people might not be big fans of mixing test data with application code. But, as long as it is clear that the data is meant as an example and not for production use, it can be a useful technique.

For example, let's say you're writing a GitHub client, and you're defining the Repo struct:

# github/repo.ex
defmodule GitHub.Repo do
  defstruct [
    :id,
    :name,
    :full_name,
    :description,
    :owner_id,
    :owner_url,
    :private,
    :html_url,
    :url
  ]

  @typespec t :: %__MODULE__{
    id: pos_integer(),
    name: String.t(),
    full_name: String.t(),
    description: String.t(),
    owner_id: pos_integer(),
    owner_url: String.t(),
    private: boolean(),
    html_url: String.t(),
    url: String.t()
  }
end

Here, we define the GitHub.Repo struct and document the keys type with typespec. While this provides a lot of information, extra documentation can help readers understand the data's nuances.

In this example, can you tell the difference between the name and full_name? Or url and html_url? It's hard to tell, right?

We can make it clearer. Let's add an example of the values:

# github/repo.ex
def example do
  %GitHub.Repo{
    id: 1296269,
    name: "Hello-World",
    full_name: "octocat/Hello-World",
    description: "This your first repo!",
    owner_id: 1,
    owner_url: "https://api.github.com/users/octocat",
    private: false,
    html_url: "https://github.com/octocat/Hello-World",
    url: "https://api.github.com/repos/octocat/Hello-World"
  }
end

Defining the example/0 function, which returns an example of the data structure, can be useful in various ways.

For example, if you're exploring the code in IEx (the Elixir interactive shell), you could invoke the function to quickly experiment with a complex function call. Livebook material (interactive documents that allow users to run code) could use these functions to show an example of the data shape. In production, the example values could be used as hints for form fields.

Customize Key Values of the `example` Function

One of the main purposes of an example function, however, is to create data for tests. You can make the example function even more useful by allowing the caller to customize the key values. Here's an example:

def example(attributes \\ []) do
  struct!(
    %GitHub.Repo{
      id: 1296269,
      name: "Hello-World",
      full_name: "octocat/Hello-World",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/Hello-World",
      url: "https://api.github.com/repos/octocat/Hello-World"
    },
    attributes
  )
end

We use Elixir's struct!/2 kernel function to build structs dynamically. The great thing about struct! is that it fails if any unexpected keys are set. Now we can invoke the example/1 function and customize the data in any way we want:

test "renders anchor tag to the repository" do
  repo = GitHub.Repo.example(html_url: "http://test.com")

  assert hyperlink_tag(repo) =~ "<a href=\"http://test.com\""
end

In the example above, we can easily create a GitHub.Repo and customize its html_url key for our test.

Similarities to Test Factories for Elixir

"Wait, isn't this just factories again?" you might be wondering. And you're right! It's similar to the factory mechanism found in other libraries. While this can make the functions easy to navigate and localize using your editor, if you rename the GitHub.Repo module, you'll need to find and replace a bunch of tests. However, modern editors are usually powerful enough to handle the task in a single command, so this shouldn't be a big issue.

Another interesting aspect of providing data examples in your struct modules is that you don't need to organize your factory files, as they are organized together within your application code.

The popular ExMachina factories, your own helper functions that use your application's rules, and data example functions in your structs are all examples of factory pattern implementations.

Using ExMachina

Using ExMachina to create your factories helps you separate test data from production code and gives you convenient functions. For example, when you define a factory using ExMachina, you can use that definition to generate test data with different strategies, like:

build to generate in-memory structs
insert to load data in a database
build_list to generate multiple items in a list
string_params_for to create map params with keys as strings like you would receive in a Phoenix controller

These are a few examples of functions that ExMachina can offer — and it has more! The convenience of these functions is debatable, though. For example, having the insert function so conveniently available could make you unnecessarily insert things into a database. And how often do your controller parameters' keys and value formats match the schema attributes generated by the string_params_for to make it really worth it? However, these convenient functions do the job for simple cases and offer a foundation for your entire test suite.

Up Next: Elixir Libraries for Test Data

Now that you understand the fundamental techniques for generating test data in Elixir, you should be able to do this for your own project without much trouble.

In the third and final part of this series, we'll dive into some Elixir libraries for your test data, including ExMachina, ExZample, Faker, and StreamData.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Test Factories and Fixtures for Elixir

Roy Tomeij — 2023-02-28T00:00:00+00:00

Writing tests is an essential part of any Elixir developer's routine. We're constantly chasing knowledge on how to write better tests, improving their speed and readability.

In this three-part series, we'll explore test data generation in Elixir. Whether you are a mid-level or senior-level Elixir developer, this series will provide valuable insights to help improve the testing process for your projects.

Let's start by digging into where test factories come from and why they are so popular in Elixir.

What are Test Factories in Elixir?

Test factories are functions for generating data, commonly used in the :test environment.

Test factory libraries like ExMachina are widely used among Elixir developers. If you look at the hex.pm stats, you'll see that ExMachina downloads aren't far behind the most popular Elixir database library: Ecto.

They allow you to create complex data structures using a very convenient interface that requires few inputs. Here's an example using ExMachina:

iex> user = ExMachina.insert(:user)
%User{
  id: "usr_xlkt"
  name: "Abilidebob"
  accounts: %Account{
    id: "acc_xktt"
    #....
  }
  #...
}

With a few characters, I can grab an example of a :user in the system. Test factories are inspired by the factory method pattern, a design pattern that allows a caller to create objects without knowing the specific module or class of the data that will be created.

Okay, this might sound a little bit complicated! Let's look at the following ExMachina code:

user = ExMachina.insert(:user)

do_something(user)

In this example, ExMachina is using the :user atom to dispatch to a function that can create examples of %User{} structs. If you write your tests in a dynamic style, you can rename the User struct without refactoring any tests, except the factory function.

The factory pattern isn't only helpful in tests. You can also use it to create or dispatch functions from dynamic sources, such as user inputs.

For example, you could use a factory function to create an account based on a user's input. A factory function is useful here because your application's users don't know the name of your source code's modules or functions — and you should keep it that way. You need to create a mechanism that links a user's input with the correct constructor:

def new_account("hobbyist"), do: HobbyAccount.new()
def new_account("professional"), do: ProfessionalAccount.new()
def new_account("enterprise"), do: EnterpriseAccount.new()
def new_account(account_type), do: raise ImpossibleAccount, account_type

In the previous example, I built a factory function named new_account/1 that takes a string as an argument and uses pattern matching to determine which function to call. For example, if a user selects the "Hobbyist" option from a selection box, the application will receive a "hobbyist" string as the input. The new_account/1 function can use the string to dispatch the HobbyAccount.new/1 function and generate a HobbyAcccount struct.

Why Use Test Factories for Elixir?

Test factories are good options for long-term test suite maintainability because of the following:

Learnability - factories document what data structures can look like in production
Reusability - data examples generated by factories can be reused in many different tests
Productivity - developers can invoke complex data examples from factories by typing a few characters instead of building new examples from scratch every time
Changeability - tests that rely on central factories always have the most up-to-date examples

These properties aren't inherently part of the factories alone. A software development team's discipline is required.

For example, if you don't provide relevant real-world examples in your factories, they will not be a source of knowledge for developers. If your factories produce values that other developers tend to override, they will more likely stop using those factories because they are not convenient anymore.

But when a factory pattern is well-maintained, it helps developers to maintain a sustainable test suite that lasts for a long time.

Now we'll turn our attention to test fixtures.

Test Fixtures in Elixir

In testing, a fixture is data that we prepare before running a test. Factories are functions that generate test data on demand.

Factories can complement fixtures — you don't necessarily have to opt for one over the other. For example, you can prepare a test fixture that uses a factory function to insert data in a database.

That's why we can sometimes use these terms interchangeably and still be understood. Often, people associate the term "fixtures" with the feature in Ruby on Rails. Ruby On Rails fixtures have the following properties:

Data is defined in YAML files, not inline in the tests.
All defined data is loaded in the database before each test run.
You can reference these fixtures through dynamic methods generated by the framework.

However, different frameworks or libraries implement test fixtures differently. After all, the main purpose of test fixtures is to provide a common starting point for your tests.

There are basically two main kinds of test fixtures: inline and implicit.

Inline Test Fixtures

The simplest inline fixture you can do in Elixir is the following:

test "creates an author profile" do
  user = MyApp.Repo.insert!(%User{id: "usr_123", name: "Abilidebob"})

  # rest of the test code
end

With a basic Repo.insert!/1, we make a database fixture for a test. That test can rely on the user with the id "usr_123" always being present in the database.

This example code assumes the test suite uses Ecto sandbox setup, which always reverts database changes made during a test after its execution.

So we don't need to worry at the beginning of the test if the user with the same id is already inserted in a previous execution. We can even use a factory function to insert the data, and it still would be correct to call it a test fixture. This fixture is explicit and inline with the tests.

Implicit Test Fixtures

Like in Ruby on Rails, it's possible to create implicit fixtures that multiple tests can share using the ExUnit.CaseTemplate test template:

# test/support/data_case.ex
defmodule MyApp.DataCase do
  use ExUnit.CaseTemplate

  # ... bunch of other code here

  setup _context do
    user = %User{id: "usr_123", name: "Abilidebob"}
    %{user: MyApp.Repo.insert!(user)}
  end
end

The code above uses the setup directive to make all tests that use this template insert a user. Then we return a map, where the key is :user, and the value is our recently created user. Any test can quickly reference this by using the key :user and grabbing it from the test context. For example:

defmodule MyApp.MyTest do
  use MyApp.DataCase

  test "hello world", %{user: user} do
    # then I use my `user` here
  end
end

ExUnit Contexts

ExUnit test contexts are a flexible mechanism where you can build things to share between multiple tests.

The biggest disadvantage of fixtures is their impact on test suite speed and test isolation if you share them too much. The speed can decrease because all the code you put in setup blocks of test templates always runs before each test, even when you don't need the data from the test context.

So be careful how much data you add and how multiple modules use the templates. Otherwise, as the speed decreases, implicit coupling between multiple tests on the same test fixtures can also increase. A small change to a test fixture shared in this way can make multiple tests fail for no obvious reason.

Sharing Bypass Instances Between Multiple Tests

Developers favor factory libraries over Ruby On Rails fixtures because they'd rather have testing data explicitly invoked from tests than implicit data generated far from where it is needed. In other words, they prefer inline test fixtures.

Explicitness contributes to better long-term maintenance. Implicit test fixtures are better for things that are very cheap to build and have a very general purpose, and low coupling with testing specifics.

A good example is the Bypass instance:

# test/support/data_case.ex
defmodule MyApp.GithubIntegrationCase do
  use ExUnit.CaseTemplate

  # ... bunch of other code here

  setup _context do
    bypass = Bypass.open()
    client = GitHub.new(endpoint: "http://localhost:#{bypass.port}/")
    %{bypass: bypass, client: client}
  end
end

In the example above, I prepare a test template for my GitHub client integration tests. The setup code creates a fake HTTP server using Bypass. The GitHub client uses this server to make HTTP requests during the tests using the client settings.

Now, all tests that use MyApp.GithubIntegrationCase can use the :bypass and :client keys to pull the fake server and its settings, respectively, from the test context.

Here is how these fixtures might be used in a test:

defmodule MyApp.MyTest do
  use MyApp.GithubIntegrationCase

  test "user registration", %{bypass: bypass, client: client} do
    Bypass.expect(bypass, :post, "/users") do
      # ... setup expectations and return a success response
    end

    # Use the `client` tag to make HTTP requests with the `GitHub` client
    {:ok, user} = GitHub.create_user(client, name: "Abilidebob")

    # Assert that the user is correct
    # ...
  end
end

If you've worked with other frameworks, such as Ruby on Rails, you might be surprised that Elixir doesn't have a popular fixture library.

I believe that's because the ExUnit test context can share test fixtures pretty easily. The fixture pattern is usually discouraged due to its impact on test suite speed and maintainability.

Test Factories and Your Elixir App's Rules

In his excellent post Towards Maintainable Elixir: Testing, Saša Jurić provides valuable tips on how to maintain a test suite. The focus of the post is on increasing your confidence in tests and reducing test overlap. There is no mention of test factories. Instead, Saša uses his own application interface to prepare test data. The reason he avoids test factories is because of their biggest problem: they bypass your application's rules.

When you set up a test factory for your test suite, the data examples it generates are completely detached from your application code's rules.

For example, let's say you have a User entity with an is_author flag, but that flag is only set to true if the user has one author profile. Here's an example of how the production code would look:

def create_author_profile(user) do
  author_profile = build_author_profile(user)
  user = Ecto.Changeset.change(user, %{is_author: true})

  Multi.new()
  |> Multi.update(:user, user)
  |> Multi.insert(:author_profile, author_profile)
  |> Repo.transaction()
end

Let's say that you now want to create the :user and :author factories for your test factory.

Here's an example using the ExMachina library:

def user_factory do
  %User{name: "Abili"}
end

def author_factory do
  %Author{
    name: "Mr. DeBob",
    user: build(:user, is_author: true)
  }
end

In the example above, the factories are straightforward. They build structs and assign values that make sense. However, have you noticed the is_author: true part? That tiny part is actually the biggest problem with factories: we have to duplicate our business rules!

Imagine having to remember to replicate every single aspect of multiple data modifications in your factories. Not only does this add extra code that needs to be maintained, but it can also mislead developers into trusting invalid data that your app cannot produce.

The invalid data generated by factories can lead you to write unnecessarily defensive code. For example, if someone forgets to include is_author: true in the factory, you might write code like this:

if user.is_author || Profiles.find_author(user_id) do
# logic for users that are author here

In the previous example, a check is written for the is_author flag. It is assumed the flag isn't always reliable — after all, the factory generates authors with is_author: false users. As a result, we decide to make a query in the database to double-check if the user is an author.

As you can see, factories that are decoupled from application rules aren't always perfect and can cause problems if they are not properly maintained.

Despite their issues, factories are a popular pattern for generating test data in Elixir. They offer convenient functions to build complex data structures, and you can invoke them on demand.

However, using factories adds an extra layer of maintainability, an extra worry to put in your team's minds. The worry is usually negligible when your application code is small, but can become problematic as your business and code grow more complex.

Up Next: Generating Data Functions in Your Elixir App

In this first part of our three-part series, we introduced test factories and test fixtures for Elixir. We then explored the biggest issue with test factories: the fact that they bypass your application's rules.

In part two, we'll explore how you can combat this by creating data generation functions using your application's rules.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Under the Hood of Ecto

Roy Tomeij — 2023-02-14T00:00:00+00:00

Ecto is a toolkit for mapping database objects to Elixir structs and provides a unified interface to manipulate that data.

In this post, we will dive into the internals of Ecto — its major components, their functions, and how they work. In doing so, we'll demystify some of the apparent magic behind Ecto.

Let's get going!

Ecto's Modules

Ecto is made up of four major modules — Repo, Query, Schema, and Changeset.

We'll look at each in turn. Let’s start with the Repo module.

Repo Module

If you use Ecto with a database (like most users out there), Repo is the heart of Ecto. It binds everything together and provides a centralized point of communication between a database and your application. Repo:

maintains connections
executes queries against a database
provides an API to write migrations that interact with the database

Let's get started with Repo. Simply call use Ecto.Repo inside your Repo module. If you use mix phx.new to generate your Elixir project, this is done automatically for you.

# lib/my_app/repo.ex
defmodule MyApp.Repo
  use Ecto.Repo,
      otp_app: :my_app,
      adapter: Ecto.Adapters.Postgres

These few lines of code define the repo. Putting it under the Supervision tree inside application.ex gives you access to a whole set of functions provided by Repo to interact with a database. Again, this is code that is generated for you when using Phoenix:

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Start the Ecto repository
      MyApp.Repo,
      # Other Children...
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

With the few lines of code above, you get the following:

Access to the full Ecto.Repo API included in MyApp.Repo. The most common use cases include fetching records with MyApp.Repo.all/2, inserting new records with MyApp.Repo.insert/2, and updating records with MyApp.Repo.update/2.
A Supervisor starts that keeps track of all the processes required to keep Ecto working. The Supervision tree initializes the adapter (Ecto.Adapters.Postgres, in this case), which is responsible for all communication with the database. The Postgres adapter, in turn, starts a connection pool to your database using the DBConnection library.
A query planner starts that's responsible for planning and normalizing a query and its parameters. It also keeps a cache of all planned queries in an ETS table. We will learn more about this when we get to the Query module.

Monitor Queries Sent to Ecto from Your Elixir Application

In addition, Ecto also automatically publishes telemetry events that can be monitored. For example, to monitor statistics for all the queries sent to Ecto, you can subscribe to the [:my_app, :repo, :query] event with telemetry.

Then, each time a query is performed, this event triggers some query metadata that includes the time spent executing the query, retrieving the data from the database, and more.

For more details, see this full list of Ecto telemetry events.

There are many options available to configure the Repo or the adapter as per your needs, but that's out of the scope of this post. Let's just take a very quick look at how you can monitor queries with AppSignal.

Instrumenting Ecto Queries with AppSignal in Your Elixir App

AppSignal automatically instruments Ecto so you can get insights into Queries running in your Phoenix or Plug applications. Make sure the :otp_app configuration option matches your app’s OTP app name, and you’re all set!

Here's an example of how an Ecto query will look in AppSignal:

Query Module

The Query module provides a unified API to write database-agnostic queries in Elixir. Note that building database queries with functions provided by the Ecto.Query module does not result in the queries being executed.

These functions return a query in the form of an Ecto.Query struct. Nothing is actually sent to the database until the built %Ecto.Query{} is passed to one of the functions provided by the Repo module.

As an example, let’s see a simple query that selects all users above 18 years of age:

age = 18
query = from u in "users",
                where: u.age > ^age,
                select: u.name

Type this into an IEx console and you will see that it creates a struct like this:

#Ecto.Query<from u0 in "users", where: u0.age > 18, select: u0.name>

You can also print it as a full map to see everything inside it:

iex(9)> IO.inspect(query, structs: false)
%{
  __struct__: Ecto.Query,
  aliases: %{},
  assocs: [],
  combinations: [],
  distinct: nil,
  from: %{
    __struct__: Ecto.Query.FromExpr,
    as: nil,
    file: "iex",
    hints: [],
    line: 40,
    params: [],
    prefix: nil,
    source: {"users", nil}
  },
  group_bys: [],
  havings: [],
  joins: [],
  limit: nil,
  lock: nil,
  offset: nil,
  order_bys: [],
  prefix: nil,
  preloads: [],
  select: %{
    __struct__: Ecto.Query.SelectExpr,
    aliases: %{},
    expr: {{:., [], [{:&, [], [0]}, :name]}, [], []},
    fields: nil,
    file: "iex",
    line: 40,
    params: [],
    subqueries: [],
    take: %{}
  },
  sources: nil,
  updates: [],
  wheres: [
    %{
      __struct__: Ecto.Query.BooleanExpr,
      expr: {:>, [], [{{:., [], [{:&, [], [0]}, :age]}, [], []}, {:^, [], [0]}]},
      file: "iex",
      line: 40,
      op: :and,
      params: [{18, {0, :age}}],
      subqueries: []
    }
  ],
  windows: [],
  with_ctes: nil
}

This is much more interesting — it shows exactly how the simple query is represented internally in Ecto. Ecto.Query.FromExpr contains details about the table we are querying (users).

ASTs in the Query

The other two expressions we see in the query are much more complex, but this is something that the adapters understand and convert to the query. If you look closely, they are ASTs.

Note: If you are interested in learning more about ASTs, check out An Introduction to Metaprogramming in Elixir.

Let's see what the code looks like for this expression:

iex> Macro.to_string({:>, [], [{{:., [], [{:&, [], [0]}, :age]}, [], []}, {:^, [], [0]}]})
"&0.age() > ^0"

This is our condition in where, just normalized into terms the adapters understand.

The adapter does the final translation of the query to actual SQL that the database understands. Note that while we usually write SQL here, the adapters don't need to work with SQL databases only — some adapters work just as well with no-SQL databases. Query generation and all database communication are clearly separated from Ecto's core.

If you want to explore this further, try building out some complex queries with joins, subqueries, windows, etc., and see how they are represented internally — it is a great way to learn how abstractions are made inside Ecto.

Finally, this query struct is converted to a SQL statement by the adapter:

iex> Ecto.Adapters.SQL.to_sql(:all, MyApp.Repo, query)
{"SELECT u0.\"name\" FROM \"users\" AS u0 WHERE (u0.\"age\" > $1)", [18]}

Back to Erlang's ETS Table

Back in the section about the Repo module, we created an ETS table when we started the Repository in our application.

Now that ETS table comes into play. When a query is executed multiple times, the query is only prepared the first time.

Note: You can learn more about PREPARE in the context of Postgres.

It is then cached inside that ETS table and fetched from there for all subsequent calls. To see the caching in action, check this out (notice the :cached in result, which signals that this query has been cached):

iex> MyApp.Repo.all(query) # This puts the query in the cache

# Trying to prepare the query again gets a cached version
iex> Ecto.Adapter.Queryable.prepare_query(:all, MyApp.Repo, query)
{{:cached, #Function<41.44551318/1 in Ecto.Query.Planner.query_with_cache/8>,
  #Function<42.44551318/1 in Ecto.Query.Planner.query_with_cache/8>,
  {5122,
   %Postgrex.Query{
     ref: #Reference<0.3269063957.2912157697.165169>,
     name: "ecto_5122",
     statement: "SELECT u0.\"name\" FROM \"users\" AS u0 WHERE (u0.\"age\" > $1)",
     param_oids: [20],
     param_formats: [:binary],
     param_types: [Postgrex.Extensions.Int8],
     columns: ["name"],
     result_oids: [1043],
     result_formats: [:binary],
     result_types: [Postgrex.Extensions.Raw],
     types: {Postgrex.DefaultTypes, #Reference<0.3269063957.2912288771.132368>},
     cache: :reference
   }}}, [18]}

Note that this doesn’t cache the result, only the prepared statements. Prepared statements give a large performance advantage, especially for complex queries. From the Postgres docs:

Prepared statements potentially have the largest performance advantage when a single session is being used to execute a large number of similar statements.

The performance difference will be particularly significant if the statements are complex to plan or rewrite, e.g., if the query involves a join of many tables or requires the application of several rules.

The next important module in Ecto is the Schema. Let's take a look at it now.

Schema Module

You can use Ecto without schemas and it works just as well (as we saw above when we referenced the table names directly).

The Schema module is responsible for defining and mapping a record's attributes (fields and associations) from a database table to an Elixir struct.

To create a schema, we write use Ecto.Schema at the top of our module and use the schema DSL.

For example:

defmodule MyApp.Organization do
  use Ecto.Schema

  schema "organizations" do
    field :name, :string
  end
end

defmodule MyApp.User do
  use Ecto.Schema

  schema "users" do
    field :name, :string
    belongs_to :organization, MyApp.Organization
  end
end

The use statement includes several utility functions and macros inside the module and sets some default module attributes required for Ecto to gather data from the Schema.
The schema macro then updates some of those attributes to mark that this is a persisted schema (there is also another embedded_schema macro to deal with non-persisted schemas) and sets some other defaults, like the primary key.
The field and belongs_to inside the schema block then put those fields in the module attributes (for type validation), and add the fields to the struct defined by the module.

The Ecto.Schema behavior exposes some methods inside the schema to fetch field details. For example:

iex(62)> MyApp.User.__schema__(:source)
"users"
iex(63)> MyApp.User.__schema__(:fields)
[:id, :name, :organization_id]
iex(64)> MyApp.User.__schema__(:primary_key)
[:id]
iex(65)> MyApp.User.__schema__(:associations)
[:organization]
iex(66)> MyApp.User.__schema__(:association, :organization)
%Ecto.Association.BelongsTo{
  field: :organization,
  owner: MyApp.User,
  related: MyApp.Organization,
  owner_key: :organization_id,
  related_key: :id,
  queryable: MyApp.Organization,
  on_cast: nil,
  on_replace: :raise,
  where: [],
  defaults: [],
  cardinality: :one,
  relationship: :parent,
  unique: true,
  ordered: false
}

This __schema__ function is also the entry-point for other parts of Ecto to reflect on more details about the defined schema and perform operations on it.

For example, when used as the source of a query, the repo will use schema to validate the conditions in the where clause, and cast the data returned from the database to Elixir structs. This results in much better feedback when there's something wrong.

A Schema Module In Action on an Elixir App

Let's try executing a query that has a typo to see the benefits of using a schema in action:

from u in "users",
  select: u.id,
  where: u.ages > 19

Executing this with Repo.all will throw a generic Postgrex.Error:

** (Postgrex.Error) ERROR 42703 (undefined_column) column u0.ages does not exist

    query: SELECT u0."id" FROM "users" AS u0 WHERE (u0."ages" > 19)

Let's try the same query, but this time with a schema.

from u in MyApp.Accounts.User,
  select: u.id,
  where: u.ages > 19

As expected, this also throws an error, but it now includes the line number where it happened and has a more specific Exception type:

** (Ecto.QueryError) lib/my_app/accounts.ex:109: field `ages` in `where` does not exist in schema MyApp.Accounts.User in query:

from u0 in MyApp.Accounts.User,
  where: u0.ages > 19,
  select: u0.id

This works because the query planner in Ecto can look at the schema's metadata and figure out that this field doesn't exist on the schema even before hitting the database.

Ecto also does type conversions behind the scenes when using the schema. For example, it allows us to run this query:

id = "2"
query = from u in MyApp.Accounts.User, where: u.id == ^id
Repo.all(query)

On the other hand, if you are not using a schema, a similar query will raise an exception:

query = from u in "users", where: u.id == ^id, select: u.id
Repo.all(query)

** (DBConnection.EncodeError) Postgrex expected an integer in -9223372036854775808..9223372036854775807, got "2". Please make sure the value you are passing matches the definition in your table or in your query or convert the value accordingly.

Changeset Module

The final module that we will look at today is Changeset. It provides an interface for validating and transforming data before it is written into a database.

Similarly to Query, Changeset provides a structured way to represent changes to data. It is most commonly used with Ecto schemas, but schemaless changesets are also possible when you don’t need a full-fledged schema.

Ecto.Changeset provides a comprehensive API to work with data.

The `cast/4` Changeset

Let's start by looking at the most commonly used changeset, cast/4:

iex> changeset = %MyApp.User{name: "some name"}
     |> cast(%{"name" => "", "organization_id" => "1", "foo" => "bar"}, [:name, :organization_id])
     |> validate_required(:name)

We pass initial data (the MyApp.User struct in this case) to cast followed by some parameters and the list of allowed fields.

cast figures out the type of each allowed field by looking at the schema metadata. cast then typecasts a parameter value to an allowed value, or adds an error to the changeset.

For example, here we can see that we had a value of 1 (String) as the organization_id. But from the schema, cast can figure out that organization_id is of type int and cast the value to an integer before putting the change inside the Changeset.

The `validate_required/3` Changeset

The second call in the pipeline, validate_required/3, requires the field to be present (as the name suggests). By default, it trims any strings/binaries before running validations and assumes an empty string to be blank.

Here's what is printed out when you inspect the Changeset.

#Ecto.Changeset<
  action: nil,
  changes: %{organization_id: 1},
  errors: [name: {"can't be blank", [validation: :required]}],
  data: #MyApp.User<>,
  valid?: false
>

This small summary already contains most of the information we need about the changeset. It shows what changes were made to the initial data (organization_id was set to 1), and that the changeset is invalid. It lists all the errors.

Let’s go one step further and inspect the full map.

iex> IO.inspect(changeset, structs: false)
%{
  __struct__: Ecto.Changeset,
  action: nil,
  changes: %{organization_id: 1},
  constraints: [],
  data: %{
    __meta__: %{
      __struct__: Ecto.Schema.Metadata,
      context: nil,
      prefix: nil,
      schema: MyApp.User,
      source: "users",
      state: :built
    },
    __struct__: MyApp.User,
    id: nil,
    name: "some name",
    organization: %{
      __cardinality__: :one,
      __field__: :organization,
      __owner__: MyApp.User,
      __struct__: Ecto.Association.NotLoaded
    },
    organization_id: nil
  },
  empty_values: [""],
  errors: [name: {"can't be blank", [validation: :required]}],
  filters: %{},
  params: %{"name" => "", "organization_id" => 1, "foo" => "bar"},
  prepare: [],
  repo: nil,
  repo_opts: [],
  required: [:name],
  types: %{
    id: :id,
    name: :string,
    organization: {:assoc,
     %{
       __struct__: Ecto.Association.BelongsTo,
       cardinality: :one,
       defaults: [],
       field: :organization,
       on_cast: nil,
       on_replace: :raise,
       ordered: false,
       owner: MyApp.User,
       owner_key: :organization_id,
       queryable: MyApp.Organization,
       related: MyApp.Organization,
       related_key: :id,
       relationship: :parent,
       unique: true,
       where: []
     }},
    organization_id: :id
  },
  valid?: false,
  validations: []
}

This contains much more information now.

The data and params are self-explanatory — the initial data and the params we fed to cast.

types contains additional data about the schema we are working with (fetched using the __schema__ method we saw in the previous section).

changes and errors are where it gets interesting. cast automatically converts the string organization_id to an integer because it knows that the organization_id is a numeric primary key from the association details.

What's also interesting is that it understands that "" is a blank value and inserts an error into the changeset from the validate_required call.

The database manipulation functions from Repo understand changesets and return errors if the changeset is invalid.

The Changeset API provides several other functions for validating data and database constraints, as well as dealing with associations.

These functions interact with the data and eventually update the struct that we saw above. When fed to the Repo module's functions, structs perform the eventual database operations.

Wrapping Up

In this post, we looked at the core concepts of the Ecto library:

We started with a Schema that defined our business objects mapped to database tables.
To get those objects out of the database, we used the Query API.
We then used Changesets to change those objects, and inserted or updated them in the database.
Tying everything together was the Repo module which takes inputs from all the other modules, eventually connecting to the database to fetch data or update records.

All of these modules work together to provide a structured and safe way to interact with databases.

Check out the official Ecto guide to integrate Ecto into your Elixir application. I have also included links to the source code in parts of this post, so feel free to go back and dig a little deeper.

Until next time — happy digging!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How To Instrument Your Elixir Application with AppSignal

Roy Tomeij — 2023-01-24T00:00:00+00:00

Instrumentation is an essential part of monitoring and operating an application, especially for apps heavily used in production. Even in today's everchanging technology landscape, visibility and observability still challenge developers and system administrators.

Metrics and logging are essential for monitoring and operating an application. Metrics measure an application's performance and system health, while logging records system health and application state. Instrumentation allows us to collect metrics and log events about an application's state and system health.

Thanks to instrumentation, you can gain great insights into your Elixir application.

In this tutorial, we will cover the importance of instrumentation, go over the basic concepts, and show you how to instrument your Elixir application with AppSignal.

Let's get started!

Why Instrument Your Elixir Application?

Application performance monitoring (APM) is the process of monitoring and keeping track of an application's performance. Many monitoring tools offer APM; in this article, we will focus on AppSignal.

Out of the box, AppSignal will handle error tracking, performance monitoring, and host metrics. However, to get the most out of AppSignal, we need to instrument our application to collect metrics and log events of specific interest to us.

With custom instrumentation, you can mark specific parts of an application to collect metrics and data, gaining insight into the deep inner workings of the application.

For example, we can use instrumentation to collect metrics about:

a specific function's performance
the number of times a particular function is called
the performance of a specific database query
the number of times a particular database query is called

With this information, developers can identify bottlenecks and refactor code to improve an application's performance.

Normally APM tools will not go deep into the layers of an application to collect metrics. Instead, they will collect metrics at the application level, such as the number of requests per second, the number of errors, and the average response time.

But as we've discussed, custom instrumentation allows developers to manually go deep into an application's layers to collect metrics and log particular events.

Application performance monitoring and custom instrumentation are essential for monitoring and operating an application and should be considered complementary tools.

Pre-requisites

To follow along with this article, you will need to have the following installed:

Elixir 1.10 or higher
PostgreSQL 12 or higher
Docker and Docker Compose v2 or higher

Start by cloning the RealWorld Phoenix application:

git clone https://github.com/tamanugi/realworld-phoenix.git

To get started, we need to install the dependencies and set up the database:

# Start Database
$ docker-compose up -d

# Install dependencies
$ mix deps.get

# Create and migrate your database
$ mix ecto.setup

# Start Phoenix endpoint with `mix phx.server`
$ mix phx.server

Validate that everything works by visiting localhost:4000/api/articles from your browser. You should be able to see the following JSON response:

{
    "articles": [
        {
            "author": {
                "bio": null,
                "following": null,
                "image": null,
                "username": "username"
            },
            "body": "It takes a Jacobian",
            "createdAt": "2022-11-06T17:43:38.000Z",
            "description": "Ever wonder how?",
            "favorited": null,
            "favoritesCount": 0,
            "slug": "how-to-train-your-dragon-1",
            "tagList": [
                "dragons",
                "training"
            ],
            "title": "How to train your dragon 1",
            "updatedAt": "2022-11-06T17:43:38.000Z"
        },
...
    ],
    "articlesCount": 10
}

Next, sign up for a free trial of AppSignal.

Once you have an AppSignal account, attach the AppSignal Elixir package to your application by adding the following to your mix.exs file:

# mix.exs
def deps do
  [
      {:appsignal_phoenix, "~> 2.0"}
  ]
end

Make sure you install the dependency by running mix deps.get.

Next, configure AppSignal by running the following mix command:

mix appsignal.install YOUR_PUSH_API_KEY

Note: YOUR_PUSH_API_KEY is the Push API Key for your AppSignal account. You can find your Push API Key in your AppSignal account under Settings > API Keys.

Follow the automated installation instructions and make sure you use the defaults. Then take one more step to capture the Phoenix HTTP requests. Open the endpoint.ex file and add Appsignal.Phoenix to the list of modules:

  # lib/realworld_phoenix_web/endpoint.ex
defmodule AppsignalPhoenixExampleWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :appsignal_phoenix_example
  use Appsignal.Phoenix

  # ...
end

Start the application again with mix phx.server, and make a couple of requests to the application by visiting localhost:4000/api/articles from your browser.

If the setup is successful, you should be able to see the requests in your AppSignal dashboard:

Here, we can see:

the number of requests per second
the number of errors
the average response time

As well as the default metrics, we can also see some custom metrics. Let's add custom metrics in the next section.

Custom Instrumentation for Your Elixir App

Now that we have AppSignal set up, let's implement some custom instrumentation. For this example, we will instrument the lib/realworld_phoenix_web/controllers/article_controller.ex/index/0 function to collect metrics on the number of times the function is called and the average response time.

Key Instrumentation Concepts

Before we proceed any further, let's quickly go over some key concepts:

Instrumentation is the process of adding observability code to our application. This code will collect metrics and log events of specific interest to us.
Trace: The record of the paths taken by a single request through the application. Tracing makes debugging less daunting by breaking down a request as the application processes it.
Span: Spans are the building blocks of traces, representing a single unit of work or operation within a system.

There are two ways to instrument our application. The first uses helper functions, and the second uses function decorators. We'll review both approaches, starting with the helper function approach.

Instrumentation Using Helper Functions for Your Elixir App

Using decorators to add custom instrumentation is relatively easy. However, there are cases where we might need more flexibility with instrumentation — for example, when we want to trace specific parts of a function.

Using the index/2 function as an example, we can use helper functions to add custom instrumentation. Open the lib/realworld_phoenix_web/controllers/article_controller.ex file and update the code to look like this:

# lib/realworld_phoenix_web/controllers/article_controller.ex
def index(conn, params) do
  # Add a delay to the function
  slow()

  keywords =  Appsignal.instrument("keywords_map", fn ->
    for {key, val} <- params, do: {String.to_atom(key), val}
  end)

  keywords = Appsignal.instrument("keywords", fn ->

      case Guardian.Plug.current_resource(conn) do
          nil -> keywords
          user -> keywords |> Keyword.put(:user, user)
      end
  end)

  articles = Appsignal.instrument("articles", fn ->
      Articles.list_articles(keywords)
      |> Articles.article_preload()
  end)

  render(conn, "index.json", articles: articles)
end

Using the Appsignal.instrument/2 function, we can wrap specific parts of the code and add events to the span. This allows us to break down the transaction into smaller parts and see where an application is spending most of its time.

Appsignal.instrument/2 allows us to instrument a function, taking three parameters:

name of the function
category (this can be left empty)
the function we are trying to instrument itself

Instrumenting Your Elixir App with the Function Decorator

The main advantage of function decorators is that we can instrument our application without modifying the code. For this app, we can use Appsignal.Instrumentation.Decorators.

Open the lib/realworld_phoenix_web/controllers/article_controller.ex file and add the following to the top of the file:

# lib/realworld_phoenix_web/controllers/article_controller.ex
use Appsignal.Instrumentation.Decorators

Also, to make this example more interesting, let's add a delay to the index/0 function. This will allow us to see the impact of the delay on the response time:

# lib/realworld_phoenix_web/controllers/article_controller.ex
def index(conn, params) do
  # Add a delay to the function
  slow()
  keywords = for {key, val} <- params, do: {String.to_atom(key), val}

  keywords =
    case Guardian.Plug.current_resource(conn) do
      nil -> keywords
      user -> keywords |> Keyword.put(:user, user)
    end

  articles =
    Articles.list_articles(keywords)
    |> Articles.article_preload()

  render(conn, "index.json", articles: articles)
end

# Decorate this function to add custom instrumentation
@decorate transaction_event()
defp slow do
  Enum.random(1000..6000)
  |>:timer.sleep()
end

Now that we have added the decorator, let's make a couple of requests to our application. You can do this by visiting localhost:4000/api/articles from your browser.

After refreshing the articles a couple of times, you should be able to see the following in your AppSignal dashboard:

We can see that the index/0 function was called and is reporting a mean of 5.1 seconds. If we click on the dashboard, we can check out the transaction details:

Here, there's information about how much time the transaction spent calling the database, parsing the route, etc. Because we added custom instrumentation, we know that the slow/0 function is the bottleneck.

Helper Vs. Decorator Functions

When it comes to choosing between decorators and helper functions, it really depends on your use case. Both approaches have their advantages and disadvantages.

Decorators are great for adding instrumentation to a function without having to modify the code. However, we can't add instrumentation to specific parts of the function and that might limit the amount of information we can gather. Yet, in cases where functions are fairly simple, decorators are a great way to add instrumentation without impacting on code.

Helper functions, on the other hand, allow us to add instrumentation to specific parts of a function. With decorators, we can only add instrumentation to an entire function. But with helpers, we can add another layer of granularity to dig deeper into potential bottlenecks in our application. Going back to our example above, we were able to break down the index/0 function into smaller parts and see where the application was spending most of its time.

However, helper functions require us to modify the code, which might not be ideal in some cases and can add complexity.

Most importantly, consider:

your use case
the amount of information you want to gather
how much granularity you need in your instrumentation.

In most cases, helper functions will be the best approach, as they can add a significant amount of granularity to your trace.

Wrapping Up

In this article, we went over the basics of adding instrumentation to an Elixir application. We learned how instrumentation can help us uncover bottlenecks and improve an application's performance. We also saw how AppSignal can help us aggregate and visualize the data we collect.

With AppSignal, we have a single tool that allows us to collect different data types, and explore the data to look for bottlenecks, issues, and potential correlations between different parts of a system.

Instrumentation is a powerful tool that will help you understand the inner workings of your application. You will be armed with the knowledge to resolve bottlenecks and catch potential issues before they become a problem.

Happy instrumenting!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Debugging and Tracing in Erlang

Roy Tomeij — 2023-01-10T00:00:00+00:00

In part one of this series, the debugging tools we used — Elixir's IO.inspect/2, IEx.pry, and dbg/2 — required us to insert changes into code directly. Outside the development environment, you probably won't (and shouldn't) have access to your code.

Erlang has a few tools to debug code at runtime: :debugger, :dbg.tracer, and :observer.

First, let's look at the :debugger, a graphical debugging tool.

Let's get going!

Debugging Runtime with `:debugger` in Erlang

Side note: While :debugger has a graphical interface, Erlang also provides :dbg. That's a debugging tool as well, but it uses a text-based interface you can run directly on IEx to achieve the same results.

The very first thing that we need to do is to start the debugger. To do that, we simply call :debugger.start/0:

iex> :debugger.start()

We'll be presented with the debugger graphical interface:

However, to start debugging, we need to tell Erlang's interpreter where to stop.

We can interact with the interpreter using the :int module. We'll use two functions: :int.ni/1 and :int.break/2.

:int.ni/1 tells the interpreter to go over all nodes and interpret the given module, while :int.break/2 adds a breakpoint in the given line:

# Tell the interpreter to interpret the given module
iex> :int.ni(Messenger.IExClient)
{:module, Messenger.IExClient}
# Add a breakpoint at line 78
iex> :int.break(Messenger.IExClient, 78)
:ok

Now, when we try to call the msg/2 function, our code will pause the execution, and the debugger interface will update:

Clicking twice on the breakpoint will open another window, so we can see the exact line of code where execution is paused. In this same window, we can also view all variables in the current execution context (like we did with IEx.pry).

As this is an Erlang tool, it will show the variables from Elixir code in a weird way. Erlang doesn't allow variables to be reassigned. The Elixir compiler has a trick to allow that: variables have versions.

Let's take the following Erlang snippet:

Name = "Jake",
Name2 = string:uppercase(Name).

Note how we have to create another variable to store the result of string:uppercase(Name). We can write this code in Elixir as:

name = "Jake"
name = String.upcase(name)

Behind the scenes, the compiler will automatically version this variable for us.

_name@1 = "Jake"
_name@2 = String.upcase(_name@1)

And that's how we see Elixir variables in the debugger window.

We can add breakpoints at any point in our code and inspect what happens on function calls, loops, recursive functions, etc. And the good thing is that we don't need to modify any lines of code as we did with IO.inspect, IEX.pry, and dbg/2 — but we still need the code to be available.

In the next section, we'll learn about tracing: a way to debug applications at runtime without modifying or having access to the source code.

Tracing (Distributed) Elixir Applications

So far, we've learned how to inspect and debug code and modules. However, sometimes we need to go even further.

Tracing an application means you can track an event starting in one function and follow its execution all the way down into the execution chain — even across the network.

Erlang provides different tools to do that. We'll look into :dbg.tracer (which is text-based) to learn the basics of tracing. Then, we'll use Trace Overview in :observer (a graphical interface) to trace applications.

Traces with `:dbg.tracer`

Because we won't use the source code to tell the debugger where to stop, to get the tracer working, we need to set up a few things:

Start Erlang's debugger
Start the tracer process
Select which nodes should be monitored (when in clusters)
Tell the tracer which processes it should monitor
Tell the tracer how to detect a function call that should be traced

First, let's start the debugger and tracer processes:

# Start the debugger
iex> :dbg.start()
# Start the tracer
iex> :dbg.tracer()

For now, the tracer will only listen to the local node. We'll change this later.

To configure the process to listen to, we'll use :dbg.p/2 (p stands for process). To keep things simple, we'll listen to all processes:

# listen to all processes
iex> :dbg.p(:all, :c)

Now we need to give the tracer some function call patterns to monitor. To do that, we need function match specs.

Match specs alone require an entire post to explain. To keep it short, think of them as a regex for functions. You can match function calls with specific values and returns, which can be very useful when debugging a problem.

The match pattern we'll use here, [], says that we don't care about the values passed as an argument. Using :dbg.fun2ms, you can write the pattern as a function, and it will generate the match spec for you.

To identify the function to debug, use :dbg.tpl/4. tpl stands for trace pattern local.

In this example, we'll match all GenServer callbacks that Messaging.MessengerServer implemented:

iex> :dbg.tpl(Messaging.MessengerServer, :handle_call, 3, [])
iex> :dbg.tpl(Messaging.MessengerServer, :handle_cast, 2, [])
iex> :dbg.tpl(Messaging.MessengerServer, :handle_info, 2, [])

Now, whenever we interact with the messenger functions, we'll be notified that it was called:

iex> sign in "alice"
Signed in!
(<0.159.0>) call 'Elixir.Messenger.MessagingServer':handle_call({sign_in,<<"alice">>,nil},{<0.160.0>,[alias|#Ref<0.3934582249.1729953800.172958>]},nil)
:ok
iex> msg "bob", "hello!"
Message sent!
(<0.159.0>) call 'Elixir.Messenger.MessagingServer':handle_call({send,<<"bob">>,<<"hello!">>},{<0.160.0>,[alias|#Ref<0.3934582249.1729953796.173763>]},#{inbox => [],notify_fun => nil,sent => [],user => <<"alice">>})
:ok

We don't need to edit code, load modules, or add breakpoints. The Erlang tracer detects function calls that match our rules and automatically traces them for us.

The log prints, but it is not Elixir-friendly because this tool was designed with Erlang code in mind. However, we can give the tracer a custom function to handle traces. With this function, we can translate the trace information into something that makes more sense to an Elixir developer.

This function will be called for all traces and given the trace information that should be handled, such as:

caller pid
module name
function name
function arguments
timestamp

iex> # Custom trace handler
iex> handler = fn data, trace_name ->
...>  case data do
...>    {_, pid, _, {module, fun, args}, timestamp} ->
...>      {{y, mm, d}, {h, m, s}} =  :calendar.now_to_datetime(timestamp)
...>      arg_list = Enum.map(args, &inspect/1)
...>
...>      IO.puts("#{trace_name} :: [#{y}-#{mm}-#{d} #{h}:#{m}:#{s}] (#{inspect(pid)}) #{inspect(module)}.#{fun}(#{Enum.join(arg_list, ", ")})")
...>  end
...> end

# Start tracer with the custom function
iex> :dbg.tracer(:process, {handler, "Initial Data"})
# Include timestamps on traces
iex> :dbg.p(:all, [:call, :timestamp])

Yes, the custom function is not pretty, but it does the job. We have to convert the timestamp from Erlang format to a readable string, and the arg list has to be transformed so we can read the values.

But now, the logged trace looks like it's coming from an Elixir application:

Initial Trace :: [2022-10-20 12:56:48] (#PID<18901.161.0>) Messenger.MessagingServer.handle_call({:sign_in, "bob", nil}, {#PID<18901.162.0>, [:alias | #Reference<18901.2941783407.3368878085.222214>]}, nil)

Now that we know the important concepts around tracing, let's do this in a cluster!

Remote Nodes

So far, we have worked with the local node, but our messenger application is designed to run in a cluster.

By tracing remote nodes, we can view everything that happens — from the moment we send a message in a local node to when it reaches the remote node.

The good news is that we already know most of what's required to do that. We just need to tell the tracer to monitor other nodes using :dbg.n (n stands for node).

To monitor the whole cluster, we iterate over the list of nodes and call :dbg.n/1:

iex> [Node.self() | Node.list()] |> Enum.each(&:dbg.n/1)

Putting it All Together: A Script that Runs a Tracer

Let's create a script that will automatically set up a tracer for us:

# tracer.exs

# Custom trace handler
handler = fn data, trace_name ->
  case data do
    {_, pid, _, {module, fun, args}, timestamp} ->
      {{y, mm, d}, {h, m, s}} =  :calendar.now_to_datetime(timestamp)
      arg_list = Enum.map(args, &inspect/1)

      IO.puts("#{trace_name} :: [#{y}-#{mm}-#{d} #{h}:#{m}:#{s}] (#{inspect(pid)}) #{inspect(module)}.#{fun}(#{Enum.join(arg_list, ", ")})")
  end
end

# Start the debugger process
:dbg.start()

# Start the tracer process
:dbg.tracer(:process, {handler, "Initial Trace"})

# Tell the tracer to monitor all nodes
[Node.self() | Node.list()] |> Enum.each(&:dbg.n/1)

# Tell the tracer to monitor all processes and include a timestamp
:dbg.p(:all, [:call, :timestamp])

# Tell the tracer the function pattern to monitor
:dbg.tpl(Messenger.MessagingServer, :handle_call, 3, [])
:dbg.tpl(Messenger.MessagingServer, :handle_cast, 2, [])
:dbg.tpl(Messenger.MessagingServer, :handle_info, 2, [])

To see it in action, let's start another node with ./start dbg that will just print the trace logs. Load the script above with c "tracer.exs" on iex:

$ ./start dbg
iex> c "tracer.exs"

The new node will see all messages exchanged between node0 and node1, as well as all internal calls:

# alice signs in
Initial Trace :: [2022-10-13 4:42:59] (#PID<18901.159.0>) Messenger.MessagingServer.handle_call({:sign_in, "alice", nil}, {#PID<18901.160.0>, [:alias | #Reference<18901.3988451517.2270494721.161681>]}, nil)

# bob signing in
ok :: [2022-10-13 4:43:8] (#PID<19318.159.0>) Messenger.MessagingServer.handle_call({:sign_in, "bob", nil}, {#PID<19318.160.0>, [:alias | #Reference<19318.2685074897.123797505.34702>]}, nil)

# alice sends "Hello!" to bob
ok :: [2022-10-13 4:43:42] (#PID<18901.159.0>) Messenger.MessagingServer.handle_call({:send, "bob", "Hello!"}, {#PID<18901.160.0>, [:alias | #Reference<18901.3988451517.2270494721.161694>]}, %{inbox: [], notify_fun: nil, sent: [], user: "alice"})

# bob receives "Hello!" from alice
ok :: [2022-10-13 4:43:42] (#PID<19318.159.0>) Messenger.MessagingServer.handle_info({:msg, "alice", "Hello!"}, %{inbox: [], notify_fun: nil, sent: [], user: "bob"})

# bob opens its inbox
ok :: [2022-10-13 4:43:52] (#PID<19318.159.0>) Messenger.MessagingServer.handle_call(:inbox, {#PID<19318.160.0>, [:alias | #Reference<19318.2685074897.123797505.34749>]}, %{inbox: [{"alice", "Hello!"}], notify_fun: nil, sent: [], user: "bob"})

By tweaking the trace handler function and the match specs, we can design powerful tracing scripts to see everything that happens in our application.

The `:observer` in Erlang

This is another valuable tool with a graphical interface included in Erlang. We can access it from iex by simply typing :observer.start:

There is a lot of information here, and you can even see stats from other nodes!

Trace Overview in `:observer`

As mentioned, :dbg is a text-based tool for debugging and tracing. But it's also possible to do the same thing using the "Trace Overview" in :observer:

And the steps are the same: select nodes, select processes, and add match specs.

Once you have this configured, hit Start Trace to view the trace logs in a new window:

Debugging Production Apps with AppSignal

AppSignal offers an integration for Erlang.

After you deploy your application to production, AppSignal can help monitor and debug errors from the real world too. You can set it up for your app easily and painlessly — once installed, you'll immediately gain access to the main dashboard, which includes error tracking, alongside some other great features.

Set up alerts, and you'll be instantly notified when errors occur.

You can drill down to see the root causes of errors and easily fix them from there:

Wrap Up

In both parts of this series, we put the most important Elixir and Erlang debugging tools into action (with the help of our demo project).

From Elixir's simple IO.inspect/2 to Erlang's powerful tracer, the Erlang ecosystem has tools for all kinds of debugging scenarios. We can use these tools seamlessly, whether locally or in a distributed system.

Happy debugging!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

AppSignal’s Top 5 Elixir posts in 2022

Roy Tomeij — 2022-12-21T00:00:00+00:00

With the holidays approaching, the AppSignal team is taking a mini winter break from publishing Elixir posts. We hope to bring you a lot more of them in 2023. 🍵 ❄️

In the meantime, let's look at the top 5 most liked Elixir blog posts from 2022!

Functional Programming in Elixir with Witchcraft

While Elixir is a functional programming language, it is different from most of the other popular functional languages like Haskell, Scala, OCaml, and F#.

In this article, I want to introduce you to a library called Witchcraft and show how you can use it to emulate Haskell-style programming in Elixir.

Algebraic Data Types in Elixir

Elixir is a dynamically-typed language. Types in Elixir are checked when a program runs, not when it compiles. If they don’t match up, an exception is thrown.

In statically-typed languages, types are checked during compile time. This can help us write code that is correct, understandable, and refactorable.

In this article, we'll cover Dialyzer, ADTs, and how you can solve the problem of illegal states by using them together.

A Guide to Event-Driven Architecture in Elixir

In this post, we will explore how event-driven architecture can make your app more responsive for users and decouple your modules for a better developer experience.

We will also look at several methods of implementing event-driven architecture with Elixir. Elixir is particularly good for this because of the advanced and concise message-passing APIs that it offers and BEAM's outstanding support for concurrency.

Livebook for Elixir: Just What the Docs Ordered

While initially conceived as a tool for data exploration (much like Jupyter for Python), Livebook has deservedly become a sensation in the Elixir community.

It has been fantastic to see all the wonderful ways teams are leveraging Livebook for a range of different use cases. We have seen Livebooks being used to:

Create interactive documentation for libraries.
Build onboarding material and guides.
Audit and explore potential dependencies in your app.

How to Write a Functor in Elixir

In this post, I will introduce you to a concept from functional programming called a functor. We’ll make a Functor protocol with a function called fmap that will aspire to be a better version of Enum.map.

AFK Time!

We hope you've learned something new while reading our blog this year. We know we have! In 2023, our team will continue working hard to publish new blog posts in our efforts to spread knowledge across the Elixir community.

Wherever you are in the world, we hope you enjoy a happy and healthy end of year. We hope that 2023 will be an amazing year for you, with no bugs and packed with events and adventures that are almost as sweet as a stroopwafel.

How to Cache Locally in Elixir with Nebulex

Roy Tomeij — 2022-12-13T00:00:00+00:00

In an Elixir application, you might need to access certain data frequently, which can be costly. The access time involved in retrieving data at every step can cause high latency, or even make the application crash (due to an increased workload on the database).

Caching is the best technique to store the most frequently accessed data and minimize database data retrieval, improving the overall performance of the application.

In this post, you'll learn how to use the Nebulex caching toolkit to:

cache locally in your Elixir applications
get familiar with different supported caching solutions/strategies
see how Nebulex can be a good fit for caching

Let's get going!

What Is Nebulex?

You've likely already come across some caching options for your existing Elixir application.

Here, I will introduce you to the Nebulex caching toolkit. Nebulex is an in-memory and distributed caching framework for Elixir that supports temporarily storing and accessing data in an Elixir application.

Why Use Nebulex in Your Elixir Application?

Caching can be implemented in many different ways depending on your application use case. You may need a local cache that stores data directly in your application or a distributed cache. Having a cache framework that supports a vast number of caching solutions is a plus. This is where we can leverage the Nebulex caching toolkit.

Nebulex has a flexible and pluggable architecture based on adapters. This makes it possible to integrate different caching options and also provides an easy workaround for various caching topologies, such as:

Partitioned caching: distribution of data between multiple nodes
Replicated caching: involving multiple servers, where each server has the same copy of data
Local caching: a single-server cache that resides on a single node

So Nebulex makes it easy for developers to scale their applications beyond a single node (with minimal impact on the code). You only need to choose the adapter that best suits your needs.

Check out this in-depth guide on supported caches and their adapters in Nebulex.

Now let's look at a real-world example where Nebulex proved useful.

The Problem: Fetching and Storing Data from an External API

A while ago, I was working on a payment integration system that involved fetching a transaction from an external API. The fetched transaction was then stored in the database.

We also had to confirm if the transaction could complete, so we scheduled a job using GenServer. GenServer sent a request to the external API after a certain period of time to confirm if the transaction was complete and then updated the stored transaction status.

This approach had the following shortcomings:

After every specified period, a request was sent to the external API to confirm if the transaction was completed. This occurred even when all fetched transactions stored in the database were completed.
After every specified period, a query ran to check if all transactions were complete. This approach resulted in unnecessary trips to the database and was deemed unfit when handling many transactions.

The Solution: Caching Locally with Nebulex

The application was being deployed to a single instance and would frequently access loaded transactions. So to solve the problem stated above, we used Nebulex to locally cache the fetched transactions.

This approach helped to overcome the following:

Making unnecessary trips to the external API after every specified period. It ensured requests were only made if the cache contained transactions. By default, all cached transactions were incomplete.
Making many trips to the database. Trips to the database were made only when inserting a fetched transaction and updating its status.

As mentioned earlier, Nebulex supports different caching solutions, one of which is local caching.

We'll now implement Nebulex in an Elixir application when caching locally.

Let's begin!

Add Nebulex to an Elixir Application

First, open the mix.exs file and add the nebulex dependency as shown:

defp deps do
  [
  {:nebulex, "~> 2.4"}

  ...

  ]
end

Install the Nebulex dependency:

mix deps.get

After installation, go ahead and generate your cache:

mix nbx.gen.cache -c Payment.Cache

This step involves:

Defining a cache module within the application
Setting up configurations

We'll use the name Payment.Cache to identify our cache (you can name yours as you wish). The command above will generate the Payment.Cache module defined in lib/payment/cache.ex.

#lib/payment/cache.ex
defmodule Payment.Cache do
  use Nebulex.Cache,
    otp_app: :my_app,
    adapter: Nebulex.Adapters.Local
end

This command generates the configuration Nebulex will use in config/config.exs,

#config/config.exs

config :my_app, Payment.Cache,
  # When using :shards as backend
  # backend: :shards,
  # GC interval for pushing new generation: 12 hrs
  gc_interval: :timer.hours(12),
  # Max 1 million entries in cache
  max_size: 1_000_000,
  # Max 2 GB of memory
  allocated_memory: 2_000_000_000,
  # GC min timeout: 10 sec
  gc_cleanup_min_timeout: :timer.seconds(10),
  # GC max timeout: 10 min
  gc_cleanup_max_timeout: :timer.minutes(10)

Finally, open lib/my_app/application.ex and add the Payment.Cache to the application supervision tree.

#lib/my_app/application.ex

use Application

  @impl true
  def start(_type, _args) do
    children = [

      Payment.Cache

      # ...

    ]

Setting Payment.Cache in the supervision tree starts the Nebulex process when the application starts up.

`Nebulex.Cache` Configuration in Elixir

The generated Payment.Cache module uses the Nebulex.Cache, a cache abstraction layer controlled by adapters. Nebulex.Cache expects :otp_app and :adapter as options.

:otp_app points to the Elixir application where nebulex can find the cache configuration. In our case, :my_app is specified.
:adapter - the desired adapter is configured at this point. In our case, Nebulex.Adapters.Local has been specified. It implements a local generation cache. This is a caching technique based on the age of the cached entries. It involves specifying the object's last modified date in the cache key.

The configurations specified in config/config.exs are specific to Nebulex.Adapters.Local passed in the :adapter option.

These are some of the options supported by Nebulex.Adapters.Local via the cache configuration:

:backend - the storage used for the adapter. The supported forms of storage are :ets and :shards, and the default is :ets.
:gc_interval - interval time in milliseconds for garbage collection to run, which involves deleting the oldest generation and creating a new one. Expects an integer > 0.
:max_size specifies the cache limit. Expects an integer > 0.
:allocated_memory - maximum size in bytes of the memory allocated for cache generation. Expects an integer > 0.
:gc_cleanup_min_timeout - minimum timeout in milliseconds for triggering the next clean-up and memory check. Defaults to 10_000 (10 seconds). Expects an integer > 0.
:gc_cleanup_max_timeout - maximum timeout in milliseconds for triggering the next clean-up and memory check. Defaults to 600_000 (10 minutes). Expects an integer > 0.

Query with `Nebulex.Cache` Callbacks

The Nebulex.Cache module has callbacks that can be leveraged to perform queries. Let's cache fetched transactions from an external API and manipulate cached entries using callbacks.

The implementation for manipulating cache entries is in lib/my_app/payment_cache.ex:

#lib/my_app/payment_cache.ex

defmodule MyApp.PaymentCache do
  @moduledoc """
  context for manipulating cache entries. It involves;
   - Inserting fetched transaction and incomplete transaction into the cache
   - Query for all cached transaction
   - Deleting cached transaction
  """
  alias Payment.Cache

  def insert_transaction(transaction) do
    Cache.put(transaction["id"], transaction)
  end

  def insert_all_transactions(transactions) do
    transactions
    |> Enum.map(fn transaction ->
      {transaction.id, %{id: transaction.id, status: transaction.status}}
    end)
    |> Cache.put_all()
  end

  def get_transaction(id) do
   Cache.get(id)
  end

  def all_cached_transactions do
    Cache.all()
  end

  def delete_transactions(transaction_ids) when is_list(transaction_ids) do
    unless Enum.empty?(transaction_ids) do
      Enum.each(transaction_ids, fn key -> delete_transaction(id) end)
    end
  end

  def delete_transaction(id) do
    Cache.delete(id)
  end
end

Here, we create a context for manipulating cache entries related to payments. We alias Payment.Cache (the module generated above in lib/payment/cache.ex). The module will give us access to the Nebulex.Cache callbacks.

The following functions are defined:

insert_transaction/1 - takes in a fetched transaction from the external API and inserts it into the cache. Payment.Cache.put/3 is responsible for inserting the transaction into the cache.
insert_all_transactions/1 - takes in a list of transactions and inserts them into the cache by invoking Payment.Cache.put_all/.
all_cached_transactions/0 - fetches all cached entries by invoking Payment.Cache.all/2.
delete_transaction/1 - deletes a cached entry by invoking Payment.Cache.delete/2.
get_transaction/1 - fetches a cached transaction from the given key by invoking Payment.Cache.get/2.

Now, open an interactive shell in your terminal, and put the functionality into use:

iex > alias MyApp.PaymentCache

iex > fetched_transaction = %{id: 1, status: "pending"}

        %{id: 1, status: "pending"}

iex > PaymentCache.insert_transaction(fetched_transaction)

      :ok

iex > PaymentCache.get_transaction(1)

      %{id: 1, status: "pending"}

iex > another_fetched_transaction =  %{id: 2, status: "pending"}

        %{id: 2, status: "pending"}

iex > PaymentCache.insert_transaction(another_fetched_transaction)

      :ok

iex > PaymentCache.all_cached_transactions()
      # returns the ids of the cached transactions
      [1, 2]

iex > PaymentCache.delete_transaction(1)

      :ok


iex > PaymentCache.all_cached_transactions()

      [2]

iex > PaymentCache.get_transaction(1)

      nil

Schedule a Job in Elixir Using Cached Entries

Caching fetched transactions comes in handy when scheduling jobs.

#lib/my_app/payment_scheduler.ex

defmodule MyApp.PaymentScheduler do
  use GenServer

  require Logger

  alias MyApp.PaymentCache

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl true
  def init(opts) do
    {:ok, opts, {:continue, :cache_incomplete_transactions}}
  end

  @impl true
  def handle_continue(:cache_incomplete_transactions, state) do

    insert_all_transaction()

    schedule_work()


    {:noreply, state}

  end

  @impl true
  def handle_info(:update, state) do
    update_pending_transactions()

    schedule_work()

    {:noreply, state}
  end

  defp schedule_work do
    Process.send_after(self(), :update, 10000)
  end

  defp update_pending_transactions do
    all_cached_transactions = PaymentCache.all_cached_transactions()

    case all_cached_transactions do
      [] -> Logger.info("All transactions are complete")

      all_cached_transactions ->

        complete_transactions = complete_transactions(all_cached_transactions)

        Payments.update_incomplete_transactions(complete_transactions)


        # delete cached transactions after update in the db
        PaymentCache.delete_transaction(complete_transactions)
    end
  end

  defp complete_transactions(cached_transactions) do

    #fetch all complete transactions from external API
    complete_transactions = Payment.confirmed_transactions()

    Enum.map(complete_transactions, fn transaction ->
      transaction.id in all_cached_transactions
      transaction.id
    end)

  end

  defp insert_all_transaction do
    pending_transactions = Transactions.all_pending_transactions()

    case Payment.all_pending_transactions() do
      [] ->  Logger.info("There are no incomplete transactions in the database")

      pending_transaction -> PaymentCache.insert_all_transaction(pending_transactions)
    end
  end
end

Let's run through what's going on here.

When an application starts, init/1 will be invoked. Our init/1 returns {:ok, opts, {:continue, :cache all incomplete transactions}}.

This will immediately trigger handle_continue(:cache_incomplete_transactions, state). Here, all incomplete transactions are fetched from the database and cached. By invoking schedule_work/0, an updated job is scheduled to take place every 10 seconds.

The handle_continue/2 callback allows us to perform a time-consuming job asynchronously, avoiding race condition cases. For more about handle_continue/2, I suggest Sophie DeBenedetto's excellent Elixir School article, 'TIL GenServer’s handle_continue/2'.

In schedule_work/0, an :update message notifies the parent process (the GenServer) that there is an update to perform. The :update message is then handled in the handle_info/2 callback. At this point, we confirm if there are incomplete transactions in the cache (keeping in mind that we only cache incomplete transactions).

When the cache is empty, that means there are no incomplete transactions. So we skip sending a request to an external API confirming whether the transactions have been completed, and we update their status in the database.

Wrapping Up

In this post, we explored how to use Nebulex for caching data locally in an Elixir application.

Implementing caching in an Elixir application depends purely on your business use case. When choosing a cache toolkit, make sure it meets your needs.

Nebulex's cache toolkit supports a vast number of caching solutions and allows you to implement different topologies with minimal impact on the code.

Visit Nebulex's guide to learn more.

Happy caching!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Debugging in Elixir and Erlang: An Introduction

Roy Tomeij — 2022-11-29T00:00:00+00:00

Welcome to part one of this two-part series on debugging in Elixir and Erlang. In this post, we'll use several different tools and techniques to debug Elixir code.

First, we'll get to know the demo project I created to showcase certain tools: dist_messenger. It's an Elixir project of a distributed messaging system that can be used directly on IEx. It allows users in different nodes to send and receive messages to each other.

Then, we'll learn about the basic debugging tools of Elixir: inspect, pry and dbg.

In part two, we'll turn our attention to Erlang, putting together what we learn into a simple script to trace a process message across nodes.

Let's get going!

The Necessity of Debugging

You write a piece of code, you run it, and it doesn't execute as expected.

That is what programming looks like most of the time. Trying to figure out what's happening can not only be time-consuming, but very frustrating when we don't have the right tools.

Can you level your washing machine without a leveling tool, instead using a ruler? Of course, but it will take much more time and energy.

So, to understand what's happening in our code, it's good sometimes to take a few steps back and ask some questions, like:

Is the value of this variable correct?
Are the transformations being applied to a variable producing the expected result?
Are the messages being correctly sent from one process to another?
Is there any abnormality in resource consumption?

Both Elixir and Erlang provide tools to answer these questions. Knowing how to use the tools certainly adds a lot of value.

Getting to Know Our Demo Project

The dist_messenger project uses a combination of GenServer and :global to provide a distributed messaging system inside an Erlang cluster.

Each node in this cluster can handle one user, and its state is maintained by a single GenServer, the Messenger.MessagingServer. Users can sign in to the cluster with a unique name, which is then registered across all nodes using the :global.register_name/2 function.

For the user interface, the Messenger.IExClient provides several functions that allow users to interact with the messaging system in IEx. This module is imported by the .iex.exs script, so we can use all its functions directly on iex when running it with iex -S mix. Check the docs for .iex.exs files.

Because each node handles one user, to get the full experience of this project, we need to run multiple nodes (by running nodes with a node name and a shared secret cookie). To make things simple, the project includes a script start that will do that for us. You just need to give the name (it will run all nodes on 0.0.0.0).

You'll notice that the included .iex.exs script will automatically try to connect to nodes node0 up to node9 on localhost. So, by naming the nodes within this range, we should get a cluster.

Let's start two nodes and sign-in two users: user alice on node0 and bob on node1.

# On terminal 1
$ ./start node0

# On terminal 2
$ ./start node0

To sign in to the messaging system, you can call sign in <name> on iex:

# node0 - user: alice
iex> sign in "alice"
Signed in!

Now you can send a message to other users using msg <user> <message>:

# node0 - user: alice
iex> msg "bob", "Hello! How are you?"
Message sent!

In bob's session, we get a notification that a new message has arrived:

# node1 - user: bob
[!] New message from alice

We can use the inbox command to view all received messages:

# node1 - user: bob
iex> inbox
> from: alice
> Hello! How are you?
---

In alice's session, we can use sent to view all sent messages:

# node0 - user: alice
iex> sent
> to: bob
> Hello! How are you?
---

There are also other commands:

signout: signs out the current users from the cluster
whoami: prints the current signed-in user
last_msg: prints the last received message
editor: sends a message like msg, but it's interactive

Now that we know how our example works, let's get started with debugging!

Elixir Debugging Basics

You can use the tools in this section to:

quickly gather information about a specific variable
get the result of a function call
check a whole pipe chain

They do require changes in the source code, so we'll use them during development and tests.

Debugging with `IO.inspect/2` in Elixir

This is the easiest — and probably the first — tool that an Elixir developer will reach for to investigate issues. The IO.inspect/2 function prints the contents of an expression and returns the expression itself.

iex> name = {:name, "Jack Shephard"}
iex> IO.inspect(name, label: "Name")
# Name: {:name, "Jack Shephard"}

It returns the value of the evaluated expression so that you can use it within a pipeline. The label option is very useful when you have multiple calls to IO.inspect and need some hints to understand what you are looking at.

person = %{name: "Kate Austen"}

iex> " some text "
...> |> String.trim()
...> |> String.upcase() |> IO.inspect(label: "Here")
...> |> String.split()
Here: "SOME TEXT"
["SOME", "TEXT"]

`IEx.pry` for Elixir

While IO.inspect allows you to read the content of any variable, IEx.pry gives us a bit more insight and information on the code.

You can think of IO.pry as an interactive IO.inspect.

Let's use it in the editor/0 function and see what happens:

...

def editor do
  case Messaging.whoami() do
    {:ok, _user} ->
      recipient = IO.gets("recipient: ") |> String.trim()
      message = IO.gets("message: ") |> String.trim()

      # Pause the execution here
      require IEx; IEx.pry

      msg(recipient, message)

    _ ->
      IO.puts("Not signed in!")
  end
end

...

Note: you must use IEx in the application context in order to use IEx.pry. This means you need to start it with iex -S mix.

When you call the editor/0 (after you type in the recipient and the message), the iex session will pause execution on the line.

iex> editor
recipient: kate.austen
message: Hello, Kate!
Break reached: Messenger.IExClient.editor/0 (lib/messenger/iex_client.ex:96)

   93:         recipient = IO.gets("recipient: ") |> String.trim()
   94:         message = IO.gets("message: ") |> String.trim()
   95:
   96:         require IEx; IEx.pry
   97:
   98:         msg(recipient, message)
   99:

pry>

In this new console, you can type a variable's name to see its contents:

pry> recipient
"kate.austen"
pry> message
"Hello, Kate!"

Type continue or next to continue the execution.

`dbg/2` Debugging in Elixir

This is a new feature since Elixir 1.14. It's pretty much IO.inspect/2 with superpowers.

To demonstrate it, we'll add dbg to this demo line. It will be called whenever the server receives a message from another server.

# ...

def handle_info({:msg, from, body}, %{inbox: inbox} = state) do
  # ...

  message = {from, body}
  updated_state = %{state | inbox: [message | inbox]} |> dbg()

  {:noreply, updated_state}
end

#...

Now, when your node receives a message, the recipient's iex session will request to pry on the code:

[!] New message from jacob
Request to pry #PID<0.159.0> at Messenger.MessagingServer.handle_info/2 (lib/messenger/messaging_server.ex:82)

   79:     end
   80:
   81:     message = {from, body}
   82:     updated_state = %{state | inbox: [message | inbox]} |> dbg()
   83:
   84:     {:noreply, updated_state}
   85:   end

Allow? [Yn]

This will open an IEx.pry session, and you can check the values of all variables.

You can also use dbg at the end of a pipe to see what's happening:

[["h"], ["i"]] #=> [["h"], ["i"]
|> List.flatten() #=> ["h", "i"]
|> Enum.join() #=> "hi
|> dbg()

dbg/2 is macro-evaluated at compile time. When called, it will inject code that allows the interpreter to pause an application's execution and pry the current context. Check out 'Elixir 1.14: Better Debugging with dbg/2 and More' for an overview of dbg/2.

Debugging Production Apps with AppSignal

After deploying your app to production, AppSignal can help debug issues from the real world.

Set up AppSignal for your Elixir application, and you'll automatically gain access to an error dashboard with information about all the errors in your app.

From there, you can dig deeper for more information on individual errors, so you can fix them:

Wrap Up

This post has introduced debugging basics in Elixir using IO.inspect/2, IEx.pry, and dbg/2. Check out Three Ways to Debug Code in Elixir for some more information.

Next time, we'll focus on Erlang specifically, debugging runtime and finding out how to trace an Elixir clustered application using Erlang's :debugger.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Parser Combinators in Elixir: A Deeper Dive

Roy Tomeij — 2022-11-15T00:00:00+00:00

In our last post, we wrote a basic parser for phone numbers using Elixir. It was a bit simplistic since it didn't really respect the format phone numbers are expected to have, but it was a great start.

We'll now improve the parser to ensure we only accept phone numbers that fit the spec and make our return type an instance of structured data.

Let's dive straight in!

Formatting Parse Results in Elixir

At this point, we've got rudimentary parsers processing the main parts of a local phone number. Let's now make the parse results easier to put into a data structure.

Right now, our parser output looks like this:

iex(1)>  Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "2", "0", "-", "4", "2", " ", "8", "4", " ", "1", "0", "5"], "", %{},
 {1, 0}, 13}

In passing, you might note that the "remaining unparsed" value (third element in the tuple) is the "" empty string, meaning our parser successfully consumes the entire input string.

For starters, let's remove noise from the parse result: we're not interested in the trunk value (it's always "0" and therefore provides no information) or the separator. Nor, for that matter, are we interested in the spaces in the subscriber number. Let's wrap each of those in an ignore/2 so they get dropped from the output:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)

  separator = choice([string("-"), string(" ")])

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)

  dutch_phone_number =
    area_code
    |> concat(ignore(separator))
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

This indeed removes the trunk prefix and separator from the parse result (given they're now both ignored), but it isn't exactly straightforward to understand since all the parts are just dumped into an array:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["2", "0", "4", "2", "8", "4", "1", "0", "5"], "", %{}, {1, 0}, 13}

We should tag the results to identify which ones come from particular parsers. That sounds like a perfect fit for tag/2:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

We're now closer to our goal:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok,
 [area_code: ["2", "0"], subscriber_number: ["4", "2", "8", "4", "1", "0", "5"]],
 "", %{}, {1, 0}, 13}

These values are still a bit messy though: we want a single string value instead of a collection of digit strings. Essentially, we want to Enum.join(digits_strings, ""), which is where reduce/3 enters the picture:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> reduce({Enum, :join, [""]})
    |> tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

We're steadily moving towards our goal of having parse results that are nicely formatted and easy to convert into a data structure:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, [area_code: ["20"], subscriber_number: ["4284105"]], "", %{}, {1, 0}, 13}

We still need a little tweak: instead of the tagged values containing an array of a single element, we want to just have the element directly. That's easy. We'll just replace our use of tag/3 with unwrap_and_tag/3:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

Check it out:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, [area_code: "20", subscriber_number: "4284105"], "", %{}, {1, 0}, 13}

Creating a Struct

Let's create a PhoneNumber struct to hold our parsed data:

# lib/demo/phone_number.ex

defmodule Demo.PhoneNumber do
  alias Demo.PhoneNumber.Parser

  defstruct [:country_code, :area_code, :subscriber_number]

  def new(string) when is_binary(string) do
    case Parser.parse(string) do
      {:ok, results, "", _, _, _} -> {:ok, struct!(__MODULE__, results)}
      {:error, reason, _rest, _, _, _} -> {:error, reason}
    end
  end
end

What do we have here? After defining a struct with defstruct/1, we define a new/1 function that will accept a string input. Then it will run our fancy parser: if the parser is unable to process the string, it will return {:error, reason}.

If, on the other hand, our parser is able to process the string, we know that results will be a keyword list of the parsed components found in the phone number (such as [area_code: "20", subscriber_number: "4284105"] as shown above). And we know this will indeed be a keyword list, because we've done so much work getting our parser to format and tag the output in this manner.

All of this hard work is finally paying off: since our keyword list keys are the same as the struct's, we can create a new struct instance by simply calling struct!/2 with our parse results!

We now have the public API to parse phone numbers into a data structure. Time for a quick test in iex -S mix:

iex(1)> Demo.PhoneNumber.new("020-42 84 105")
{:ok,
 %Demo.PhoneNumber{
   area_code: "20",
   country_code: nil,
   subscriber_number: "4284105"
 }}

iex(2)> Demo.PhoneNumber.new("foobar")
{:error, "expected ..."}

Writing Custom Combinators Using Elixir

We've made great strides so far: our parser now returns a data structure! This will be extremely helpful down the line when the data it contains needs to be validated or otherwise processed.

There's still some work ahead of us, though: aside from the fact that it doesn't yet parse international numbers, it also doesn't check for number lengths. Area codes may have 2 or 3 digits, while subscriber numbers may have 7 or 6 digits respectively.

To assist us in achieving this goal, we'll be writing more advanced and powerful combinators. As you may recall from the previous article, combinators are parsers that can be combined into bigger and better parsers. We'll be using NimbleParsec to define a complex parser from smaller and simpler ones.

Since we want the parser behavior to vary according to provided arguments, we need to define some functions. But since NimbleParsec's defparsec/3 is executed during compilation, we can't invoke a function defined in the same module.

To work around this, we'll define a Helpers sub-module and our configurable parsers there:

# lib/demo/phone_number/parser/helpers.ex

defmodule Demo.PhoneNumber.Parser.Helpers do
  @moduledoc false

  import NimbleParsec

  @doc """
  Parses a single digit.
  """
  def digit(combinator \\ empty()) do
    combinator
    |> utf8_string([?0..?9], 1)
  end

  @doc """
  Parses `count` digits.

  For example, `digits(3)` would parse 3 digits.
  """
  def digits(combinator \\ empty(), count) do
    combinator
    |> duplicate(digit(), count)
    |> reduce({Enum, :join, [""]})
  end

  @doc """
  Parses a phone number area code.

  Since phone number area codes can only be 2 or 3 digits in length,
  only `2` and `3` are valid values for the `length` parameter.
  """
  def area_code(combinator \\ empty(), length) when length in [2, 3] do
    combinator
    |> digits(length)
  end
end

As you can see, we use combinator as the first function argument, defaulting to NimbleParsec's empty() combinator if none is provided. This enables us to compose combinators where appropriate, without having to use concat/2 as a crutch.

Updating the Parser with Functions

Let's update our parser to use these new functions:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  # ...

  area_code_with_trunk_prefix =
    ignore(trunk_prefix)
    |> area_code(2)
    |> unwrap_and_tag(:area_code)

  # ...

  subscriber_number =
    choice([digit(), ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:subscriber_number)

  dutch_phone_number =
    area_code_with_trunk_prefix
    |> concat(ignore(separator))
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We've renamed area_code to area_code_with_trunk_prefix so it won't get confused with the area_code imported from the new Helpers module. Also take note that digit has become digit(), as it's now a function rather than a combinator.

Our code still works, but we've yet to fix the subscriber number parser: if the area code is 2 digits, then the subscriber number should be 7 digits. Conversely, if the area code is 3 digits long, the subscriber should contain only 6 digits. So let's now refactor to have a variable-length subscriber_number combinator along with a configurable local_number combinator:

# lib/demo/phone_number/parser/helpers.ex

defmodule Demo.PhoneNumber.Parser.Helpers do
  @moduledoc false

  import NimbleParsec

  # ... various combinator definitions ...

  def subscriber_number(combinator \\ empty(), length) when length in [6, 7] do
    combinator
    |> digit()
    |> times(ignore(optional(string(" "))) |> digit(), length - 1)
    |> ignore(optional(string(" ")))
    |> reduce({Enum, :join, [""]})
  end

  def local_number(combinator \\ empty(), area_code_length, subscriber_number_length) do
    separator = choice([string("-"), string(" ")])

    combinator
    |> area_code(area_code_length)
    |> unwrap_and_tag(:area_code)
    |> concat(ignore(separator))
    |> concat(subscriber_number(subscriber_number_length) |> unwrap_and_tag(:subscriber_number))
  end
end

There's quite a bit going on here, so let's take a look.

Side note: if you ever get stuck when writing sub-parsers, you can create entry points to only test that portion of your code. For example, by adding defparsec(:subscriber_number, subscriber_number(7)) at the bottom of Demo.PhoneNumber.Parser, you can run iex -S mix and test the parsing of subscriber numbers with Demo.PhoneNumber.Parser.subscriber_number("65 23 125").

The subscriber number is defined as (mandatorily) starting with a digit, followed by an optional space and then another digit, with the latter grouping repeating one time less than the desired length provided as an argument. That means that we'll end up with as many actual digits as requested via length (along with a variable number of spaces mixed in). We ignore the whitespace so it won't show up in the results, and then combine all parsed digits into a single string.

We then define a local_number combinator that will accept two lengths: one for the area code, and the other for the subscriber portion. We combine these with an ignored separator, and tag each segment. Since we know each area_code and subscriber_number combinator will yield a single result, we can use unwrap_and_tag/3 to extract the parsed result from the returned array and tag it with the provided atom.

As you may have seen, our subscriber_number combinator accepts a previous combinator as the first argument: we could have chained it directly to the previous combinator.

Why, then, are we using concat/2? Because it will, in effect, "separate" the combinator we're interested in, so that unwrap_and_tag/3 will apply only to the subscriber_number sub-parser rather than to the combinator result of subscriber_number and its prior combinator.

In addition, the Enum.join step we have in subscriber_number would run into problems without concat/2. It attempts to join the result of the previous combinator (which will include area_code: "20"), so it will crash (as Enum.join won't know how to process a tuple). Keep this in mind if you run into unexpected failures when writing your own combinators.

Rewriting the Main Parser in Elixir

With these new custom combinators available, let's go back and rewrite our main parser:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  trunk_prefix = string("0")

  local_portion =
    choice([
      local_number(2, 7),
      local_number(3, 6)
    ])

  dutch_phone_number =
    ignore(trunk_prefix)
    |> concat(local_portion)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

Nice and readable: we can easily tell that the local portion of the number is always expected to be 9 digits long, with a longer area code resulting in a short subscriber number.

We're almost done! We just need to handle the international prefix:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  # ...

  international_prefix =
    ignore(string("+"))
    |> digits(2)
    |> ignore(string(" "))
    |> map({String, :to_integer, []})
    |> unwrap_and_tag(:country_code)

  dutch_phone_number =
    choice([international_prefix, ignore(trunk_prefix)])
    |> concat(local_portion)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We simply define the international_prefix combinator using functions that are now familiar to us, and then use map/3 to ensure the result is an integer: we may want to compare it to valid ISO country codes later on.

With that in place, it's simply a matter of telling our parser that we're either:

In the international case (where our parser does not expect the trunk prefix to be present)
In a national context, where the trunk prefix should indeed be part of the phone number (but removed from the output)

Voilà, we've got our phone number parser defined in a way that's understandable and easy to maintain. Take a look:

iex(1)> Demo.PhoneNumber.new("+31 20-42 84 105")
{:ok,
 %Demo.PhoneNumber{
   area_code: "20",
   country_code: 31,
   subscriber_number: "4284105"
 }}

Next Steps: Play Around with Parser Combinators

The best way to get a good grasp on how parser combinators work is to experiment with them.

Want to give it a spin but can't think of an example? Try extending our code to parse other cases.

Write a parser for ISO 8601 datetimes such as 2019-11-14T00:55:31.820Z. And if you want to get really crazy, you can extend it to handle durations and time intervals.

You can also try your hand at writing a recursive parser: there's an example in NimbleParsec's parsec/2 documentation to get you started.

As touched upon in the introduction to this article, NimbleParsec is more of a parser generator: take a look at Combine and try to convert the above code to make use of Combine, or come up with your own example.

The way Combine works is that you combine parser functions, and obtain a new parser function. Parser functions can then be applied via Combine.parse/3. Here's a toy example:

# the public parser
def username_and_id(string) do
  Combine.parse(string, username_and_id_parser())
end

# these are internal parsers for the combinator

# a "username and id" is simply a sequence of
# username, space, id
defp username_and_id_parser() do
  sequence([
    username(),
    ignore(string(" ")),
    id()
  ])
end

# a username consists of word characters
defp username(), do: word()

# an id is an integer
defp id(), do: integer()

Calling this function will return the (non-ignored) parse result:

iex> username_and_id("foobar 123")
[["foobar", 123]]

Ready for the big leagues? Write your own parser combinator from scratch to fully understand how everything works under the hood. Here's an example parsing a subset of SQL, written by Saša Jurić.

Wrapping Up

In the first part of this series, we briefly defined parser combinators before building a simple one in Elixir.

In this second and final part, we dived deeper into the parser combinator that we built last time and improved it. We also gave you some ideas for ways to further explore parser combinators.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Sanitize Strings in Elixir with Pattern Matching and Recursion

Roy Tomeij — 2022-11-01T00:00:00+00:00

In this post, we'll use two of Elixir's most powerful features - pattern matching and recursion - to sanitize a string by removing invalid Unicode "Specials" characters.

Let's dive straight in!

The Problem: Strings with Unicode Specials

Unicode Specials is a short Unicode block of characters allocated at characters U+FFF0-FFFF. If one of these characters is present in a string of text, that text is not correctly encoded Unicode text. Here's the full set of those characters:

0xFFF0
0xFFF1
0xFFF2
0xFFF3
0xFFF4
0xFFF5
0xFFF6
0xFFF7
0xFFF8
0xFFF0
0xFFF10
0xFFF11
0xFFF12
0xFFF13
0xFFF14
0xFFFA
0xFFFB
0xFFFC
0xFFFD
0xFFFE
0xFFFF

You might encounter incorrectly encoded strings that contain a member of the Unicode Specials block. Let's say you need a way to process those strings anyway. Maybe you have some code that reads file contents or receives string input from another external source, like a form file upload or external API. In this case, you will want to remove these invalid characters to continue processing the text.

Elixir makes this easy with the help of binary pattern matching and recursion. We'll build out a function that removes Unicode specials from a string and replaces them with the "�" replacement character, so your program can continue to process the string input. When we're done, you'll have a handy function you can use to sanitize any string and a deeper understanding of two Elixir language features that will prove useful in many other scenarios.

Let's get started!

The Solution: Sanitize Your String with Pattern Matching and Recursion in Elixir

Our goal is to produce a function - CleanString.unicode_only/2 - that takes in any string and returns that string, with Unicode Specials replaced with the "�" replacement character. Here are a few examples of our finished product in action:

CleanString.unicode_only("abc" <> <<0xFFFF::16>>)
=> "abc�"
CleanString.unicode_only("a" <> <<0xFFFF::16>> <> "bc" <> <<0xFFFF::16>>)
=> "a�bc�"
CleanString.unicode_only("abc" <> <<0xFFF2::16>> <> "def")
=> "abc�def"
CleanString.unicode_only(<<0xFFF5::16>> <> "def")
=> "�def"
CleanString.unicode_only(<<0xFFF5::16>> <> <<0xFFF4::16>> <> "def")
=> "��def"

Step 1: Break Down the Problem

Before we write any code, let's break down this problem one step at a time:

For each character in a string
Check if that character is present in the @specials list
If it is, remove it from the string
And replace it with the replacement character
If it is not present in the @specials list
Keep that character

Changing elements in place in a string is a tricky and potentially expensive operation, but Elixir gives us another way.

We can use binary pattern matching to iterate over each element in our string, recursively breaking off the "head" or first character of the string. Then, depending on whether or not that character is considered valid, we will add it to the head of a new string.

Pulling the head off a string and adding to the head of a new string are very performant tasks in Elixir--it doesn't matter how long the string is, Elixir will only ever need to remove from or add to the beginning of that string.

Let's break down our problem one more time, this time with our new strategy in place:

For each character in a string
Check if that character is present in the @specials list
If it is, add a replacement character to the head of a new string
If it is not, add that character to the head of a new string

This will result in a brand new string that contains only replacement or valid Unicode characters.

With our plan in place, we're ready to write some code.

Step 2: Define the Module in Elixir

First up, we'll define a module, CleanString, that sets a module attribute, @specials, to a list of binary strings that represent the characters in the specials block listed above:

defmodule CleanString do
    @specials [
    <<255, 240>>,
    <<255, 241>>,
    <<255, 242>>,
    <<255, 243>>,
    <<255, 244>>,
    <<255, 245>>,
    <<255, 246>>,
    <<255, 247>>,
    <<255, 248>>,
    <<255, 240>>,
    <<255, 16>>,
    <<255, 17>>,
    <<255, 18>>,
    <<255, 19>>,
    <<255, 20>>,
    <<255, 250>>,
    <<255, 251>>,
    <<255, 252>>,
    <<255, 253>>
    <<255, 254>>,
    <<255, 255>>
  ]
end

This list of binary strings is generated by taking each specials character and evaluating its hexadecimal bitstring version like this:

iex> <<0xFFFF::16>>
<<255, 255>>

We want our module to store this list of binary strings so that we can check each character of the given string against this list and remove it if it's found.

Now that we have the basics of our module in place, let's start building the unicode_only/2 function.

Step 3: Define the `unicode_only/2` Function in Elixir

The unicode_only/2 function will take in two arguments: the original string and the new "clean" string we're building.

We plan to call our function recursively. The first time, we call it with only one argument--the argument of the original string--like this:

CleanString.unicode_only("abc" <> <<0xFFFF::16>>)
=> "abc�"

In this scenario, we need to default the "clean string" second argument to an empty string. Let's define that default argument function head first:

defmodule CleanString do

  @specials [
    # ...
  ]

  def unicode_only(string, new_string \\ "")
end

Great, now we're ready to define the remainder of our function heads. Here's where Elixir's powerful binary pattern matching feature comes in.

We'll define one version of the function that pattern matches a scenario where the character at the head of the string is included in the @specials list, and one in which it isn't. Let's start with that first scenario:

@replacement_character "�"

def unicode_only(<<head::binary-size(2)>> <> tail, new_string)
    when head in @specials do
  unicode_only(tail, @replacement_character <> new_string)
end

First, we've added a new module attribute, @replacement_character, that stores our replacement character. Then, we've added a version of the unicode_only/2 function that uses binary pattern matching to extract the first two bytes of the string and set them equal to a variable, head.

Why do we need to capture the first two bytes of the string to check if the first character is part of the Unicode Specials block? This is because each individual specials character is represented by a two-byte binary string.

Let's take a closer look at how we capture those two bytes now. The magic happens in the first argument given to unicode_only/2:

<<head::binary-size(2)>> <> tail

Elixir allows you to pattern match in the function head. This means that when unicode_only/2 is called, the first argument given will be pattern matched against this definition. For example, if we call the function like this:

original_string = "abc" <> <<0xFFFF::16>>
CleanString.unicode_only(original_string)

Then the following pattern match will be executed:

<<head::binary-size(2)>> <> tail = original_string

As mentioned above, this results in setting the variable head equal to the first two bytes of the string. Meanwhile, the tail variable is bound to the remainder of the string. Then, we use a guard clause to check if that variable is included in the @specials list.

If it is, we don't want to add it to the new clean string we're constructing. Instead, we want to add the replacement character:

@replacement_character <> new_string

Now we need to perform this same operation for the next character in the original string, and the next one, and so on. This is where recursion comes in. We will call unicode_only/2 again. This time with only the tail of the original string as a first argument, and the newly constructed clean string as a second argument:

unicode_only(tail, @replacement_character <> new_string)

In this way, we recursively pull the head off the string, until the string is empty. Now, we have a function head that runs when the first two bytes of the string are included in @specials.

Let's define another version of our function that will run when the first character isn't included there.

 def unicode_only(<<head::binary-size(1)>> <> tail, new_string) do
  new_string = head <> new_string
  unicode_only(tail, new_string)
end

Once again, we use binary pattern matching in the function head. This time, we pattern match the first argument given to unicode_only/2 against the following:

<<head::binary-size(1)>> <> tail

Here, we pull off the first byte of the string and set it equal to the head variable, and we set the remainder of the string equal to tail. We only want to grab the first byte of the string here because non-specials characters are represented by a single byte, for example:

iex> <<99>>
"c"

Moving our attention back to the function body, let's talk about how this function behaves. In the scenario where the first character is not included in @specials we do want to retain that character in our new string. So we add it to the head of the new string, like this:

head <> new_string

Then, we're ready to recursively call our function again, with the remaining tail of the original string and our updated new string:

unicode_only(tail, head <> new_string)

We're almost done. We just need a way to break out of our recursive loop when the time is right. When do we want to stop recursing? When we've processed every character in the original string. When this happens, the original string will be empty, and the first argument passed to unicode_only/2 in our recursive loop will be "". Let's implement that function head now, once again using pattern matching:

def unicode_only("", new_string), do: String.reverse(new_string)

When the original string is finally empty, we don't want to call unicode_only/2 again. Instead, we'll return the reversed new string.

We need to reverse the string because we added each processed character to the head of the new string to keep our code performant. So, we added each retained or replacement character in the opposite order from their placement in the original string. Reversing the string when we're done puts everything back in the right order.

Putting it all together, we end up with this:

defmodule CleanString do

  @specials [
    <<255, 240>>,
    <<255, 241>>,
    <<255, 242>>,
    <<255, 243>>,
    <<255, 244>>,
    <<255, 245>>,
    <<255, 246>>,
    <<255, 247>>,
    <<255, 248>>,
    <<255, 240>>,
    <<255, 16>>,
    <<255, 17>>,
    <<255, 18>>,
    <<255, 19>>,
    <<255, 20>>,
    <<255, 250>>,
    <<255, 251>>,
    <<255, 252>>,
    <<255, 254>>,
    <<255, 255>>
  ]

  @replacement_character "�"

  def unicode_only(string, new_string \\ "")

  def unicode_only(<<head::binary-size(2)>> <> tail, new_string)
      when head in @specials do
    unicode_only(tail, @replacement_character <> new_string)
  end

  def unicode_only(<<head::binary-size(1)>> <> tail, new_string) do
    unicode_only(tail, head <> new_string)
  end

  def unicode_only("", new_string), do: String.reverse(new_string)
end

Wrap Up

With a set of functions implemented in around ten lines of code, we've created an elegant solution for a common, tough problem. Thanks to Elixir's powerful pattern matching functionality, which we used here to pattern match binary strings in function heads, we have a handy function for cleaning bad characters out of any string.

String sanitization is something that you may need to do in the wild, and Elixir is a perfect fit for the job. Beyond that specific scenario though, this solution demonstrates how nicely Elixir pattern matching and recursion go hand in hand.

Instead of implementing complex and difficult-to-read if statements, our recursive code flow was built entirely with pattern matching. Each scenario was mapped to a different unicode_only/2 function head.

Reach for function head pattern matching whenever you want to execute different function behavior based on different inputs, and you'll end up with clean, highly readable, and maintainable code. You can even pattern match on single characters, or groups of characters, in a binary string with the syntax you've seen here.

Taken together, pattern matching and recursion are a match made in Elixir heaven. Add these tools to your Elixir toolbox and take them out whenever you have a complex input processing problem to solve.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Parser Combinators in Elixir: Taming Semi-Structured Text

Roy Tomeij — 2022-10-18T00:00:00+00:00

The need to manipulate strings comes up quite often, whether it's to validate user-provided values or transform text into structured data that can be used programmatically.

Most often, we'll reach for regular expressions to accomplish this task, but sometimes there's a better solution to the problem: parser combinators. In this two-part article, we'll explore how they work.

Before moving on, let's define what 'parsing' is:

A parser is a software component that takes input data (frequently text) and builds a data structure.

Source: Wikipedia.

In other words, when talking about transforming text into structured data, we essentially mean parsing.

Let's get into it!

Parser Combinators: The What, How, and Why

So what are parser combinators? Well, as their name implies, they are parsers that can be combined — to make bigger, better parsers.

In effect, parser combinators enable the writing of complex parsers from simpler ones. You're less likely to make mistakes, and the parsers are a lot easier to maintain due to better readability.

In contrast, regular expressions are often a poor choice for non-trivial parsing. Writing them is error-prone, they're challenging to maintain, and their format doesn't lend itself well to documentation/explanation.

To illustrate working with parser combinators, we'll work on transforming a string representation of a Dutch phone number into a canonical representation. This can prove useful, as a web form can capture a user's phone number to later be used programmatically, for example.

The "Parser" and "Combinator" Bits: NimbleParsec

We'll be using NimbleParsec as the resulting parsers are more efficient, but other libraries such as Combine also exist and work well.

As an aside, note that it's been pointed out that NimbleParsec is more akin to a parser generator than a parser combinator: that's true, but the difference won't matter when learning how to make use of parser generators.

Parser Examples in Elixir

As mentioned before, parsing is the act of taking unstructured data (typically strings) and transforming them into structured data.

We've got some examples right in Elixir.

We can turn a string containing a URI into a data structure via URI.new/1:

iex> URI.new("https://elixir-lang.org/")
{:ok, %URI{
  fragment: nil,
  host: "elixir-lang.org",
  path: "/",
  port: 443,
  query: nil,
  scheme: "https",
  userinfo: nil
}}

iex> URI.new("/invalid_greater_than_in_path/>")
{:error, ">"}

Similarly, we can convert a string representation of an integer into an actual integer by using Integer.parse/2:

iex> Integer.parse("34")
{34, ""}

iex> Integer.parse("34.5")
{34, ".5"}

iex> Integer.parse("three")
:error

Each of these examples will accept a string and either return a parsed result (possibly with "leftover" string content, as in the Integer.parse/2 example) or an error indication.

Combinator Examples in Elixir

What if we had lists indicating how many visitors came from URIs (formatted as "URI & count" pairs)? It would be nice to have a single function to do the heavy lifting for us:

iex> parse_visit_count("https://elixir-lang.org/ 123")
{:ok, %{
  referrer: %URI{
    fragment: nil,
    host: "elixir-lang.org",
    path: "/",
    port: 443,
    query: nil,
    scheme: "https",
    userinfo: nil
  },
  visits: 123
}}

Easy, I hear you say — just do something like:

def parse_visit_count(string) do
  with [uri_string, count_string | []] <- string |> String.trim() |> String.split(),
        {:ok, uri} <- URI.new(uri_string),
        {count, ""} <- Integer.parse(count_string) do
    {:ok, %{referrer: uri, visits: count}}
  else
    _ -> :error
  end
end

That's indeed pretty straightforward. But what if there are several URI/visits pairs per line? Then the logic is already much more involved. And what if these URI/visits pairs are in reverse order (visits first, then URI)?

Using a parser combinator approach, we look at the problem differently. We've got a parser for URIs, and another one for integers. Let's write a parser for white space, then combine those three parsers into a single one for our specific use case. In pseudo-code, it would look like this:

def parse_visit_count(...) do
   ...
   |> uri_parser()
   |> whitespace_parser()
   |> integer_parser()
end

And since we're now in combinator-land, we could then use parse_visit_count and combine it with other parsers to step up the complexity. Neat, right?

Code Walkthrough: Our Parser Combinator Example

Let's work through a concrete example of using a parser combinator: we'll parse a phone number into a data structure.

The phone number we'll use for our examples is one from the Netherlands: +31 20 42 84 105.

According to Wikipedia, this number can be formatted in various ways (note we'll only be covering a subset of the possibilities within this article):

The area code ('A') is commonly separated with a dash ('-') and sometimes a space from the subscriber's number ('B'). The length of the area code for landlines is either 2 or 3 digits, depending on the population density of the area. This leaves 7 or 6 digits for the subscriber's number, resulting in a format of either 0AA-BBBBBBB or 0AAA-BBBBBB. [...] The trunk prefix '0' is dropped when prefixed by the country code: +31 AA BBBBBBBB, [...], etcetera. [...]

In other words, this phone number can also be formatted as (among others):

+31 20 4284105
+31 20-42 84 105
020-42 84 105
020-4284105
020 42 84 105

Deconstructing the Problem

From a quick look at the formats above, we can tell that the main "chunks" we need to extract are:

Country code - the "+31" prefix, which may be absent
Area code - the "020", "(0)20", or "20" value
Subscriber number - the remaining seven digits, which may be separated by spaces

Getting Started

We'll use NimbleParsec, so let's go ahead and create a new project with mix new demo and add the dependency in mix.exs:

# in mix.exs

defp deps do
  [
    {:nimble_parsec, "~> 1.2"}
  ]
end

Run mix deps.get to ensure everything's ready for us. With that in place, let's start writing our phone number parser, which we'll do within its own module:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  dutch_phone_number = string("+31 20 42 84 105")

  defparsec(:parse, dutch_phone_number)
end

Within this PhoneNumber.Parser module, we need to import NimbleParsec so we can call on its functions (such as string) more conveniently. Then, we define our parser. Right now, that consists of only parsing the exact phone number string via NimbleParsec's string/2 function (hence why the phone number value is hardcoded). And last but not least, we call defparsec/3 to "finalize" the parser and make it executable.

Let's give it a spin! Open an IEx session by typing iex -S mix in the project's root folder from within a terminal. We can then try our parser:

iex(1)> Demo.PhoneNumber.Parser.parse("+31 20 42 84 105")
{:ok, ["+31 20 42 84 105"], "", %{}, {1, 0}, 16}

iex(2)> Demo.PhoneNumber.Parser.parse("foobar")
{:error, "expected string \"+31 20 42 84 105\"", "foobar", %{}, {1, 0}, 0}

The result is a tuple containing six terms. The first is the :ok or :error tag indicating whether the provided string could be parsed.

The second value is (in the :ok case) a list of the parsed information, or (in the :error case) the error reason.

The remaining values can be ignored as we won't get into them in this article.

Parsing Local Numbers

Now that we've created a parser for an exact string, let's dig deeper into NimbleParsec's functionality by expanding the variety of phone numbers our parser will be able to process.

Local phone numbers look like this:

020-42 84 105
020-4284105
020 42 84 105

There's an area code, then a separator which can be a space or a dash. This is followed by the subscriber number, which can contain spaces to make it easier to read.

Note also that the "area code" portion consists of the trunk prefix (which is "0") followed by two or three numbers indicating the area itself.

So conceptually, we're looking at writing a parser that looks like area_code <> separator <> subscriber_number.

Let's start with the area code:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  trunk_prefix = string("0")

  area_code =
    trunk_prefix
    |> string("20")

  dutch_phone_number = area_code

  defparsec(:parse, dutch_phone_number)
end

Let's give it a whirl in iex -S mix:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "20"], "-42 84 105", %{}, {1, 0}, 3}

We can see our parser is properly extracting out the area code: it's being returned as ["0", "20"] because the area_code combinator is composed of trunk_prefix (yielding "0"), followed by string("20") (logically yielding "20"). The remainder of the input string is left unparsed (as "-42 84 105"). Success!

So what's going on? We create a trunk_prefix parser that will match the specific "0" string by using string/2.

Then, we define an area_code parser that matches this trunk_prefix followed by the "20" string. And finally, we define the parse function as the parser specified by the area_code combinator by calling defparsec/3.

We're on a roll, so let's add to our current parser so that it also captures the separator. This separator can be either a dash or a space. It sounds like a perfect match for choice/3, which will apply the first parser it finds that can parse the input.

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  separator = choice([string("-"), string(" ")])

  dutch_phone_number =
    area_code
    |> concat(separator)

  defparsec(:parse, dutch_phone_number)
end

In choice([string("-"), string(" ")]), the choice([...]) combinator will attempt to parse a "-" string. If it's not able to, it will attempt parsing a " " string. Only if both of these options fail, does the entire choice combinator fail. Conversely, if the first string("-") option succeeds, none of the subsequent parsers provided to choice will be tested.

You'll notice that we weren't able to simply pipe the two area_code and separator combinators into each other: separator doesn't accept an argument, so we can't pass in area_code. NimbleParsec does, however, make it easy to concatenate two combinators with concat/2, so that's exactly how we plug these two functions into one another. Let's check things still work as expected:

# in `iex -S mix`

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "20", "-"], "42 84 105", %{}, {1, 0}, 4}

iex(2)> Demo.PhoneNumber.Parser.parse("020 42 84 105")
{:ok, ["0", "20", " "], "42 84 105", %{}, {1, 0}, 4}

iex(3)> Demo.PhoneNumber.Parser.parse("020/42 84 105")
{:error, "expected string \"-\" or string \" \"", "/42 84 105", %{}, {1, 0}, 3}

Great! Both the dash and space separators are getting picked up, while the invalid slash yields an error.

Let's now write an overly accepting parser that we can come back to and tighten down: we want to capture all digit and space characters until "end of string". To do so, we'll use utf8_string/3 to parse digits, times/3 to get us repetition, and finally, eos/1 to represent "end of string". It looks like this:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  digit = utf8_string([?0..?9], 1)

  area_code =
    trunk_prefix
    |> times(digit, 2)

  # ...

  subscriber_number =
    choice([digit, string(" ")])
    |> times(min: 1)

  dutch_phone_number =
    area_code
    |> concat(separator)
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We define our digit combinator as a UTF8 string that contains values in the 0-9 range and is of length 1: in other words, it's a string of a single digit.

Note we've also improved the area_code combinator: instead of matching only the specific "20" area code, it will match any two digits. The side effect is that the parse results will be slightly different: ["0", "2", "0", "-", ...] instead of ["0", "20", "-", ...].

Finally, since eos() has an optional combinator argument, there's no need to wrap it with a concat() call within the pipeline.

Time for a Short Intermission

We've covered a decent amount of ground so far. Let's take a break and let all of that sink in.

Our parser is starting to take shape, but it's still not very good. We'll improve it in the next installment where we'll investigate writing custom parsers, among other topics.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Phoenix LiveView 0.18: New Special HTML Attributes

Roy Tomeij — 2022-10-11T00:00:00+00:00

Phoenix LiveView 0.18 just shipped, with lots of new goodies to make developing LiveView an even better experience.

In this post, I'll take you through a lesser-known new feature - LiveView's new special HTML attributes - and show you how to write cleaner HTML with :if, :for, and :let.

When we're done, you'll have an eloquent, ergonomic, and dynamic function component you can use to render a list anywhere in your LiveView app.

Let's dive in!

What are Special HTML Attributes in LiveView 0.18?

LiveView special HTML attributes are inspired by a Surface feature called "directives". Directives are built-in attributes that modify the translated code of a tag or component at compile time.

Directives are just one of a few new features, like component slots and declarative assigns, that LiveView has drawn from Surface. Surface is LiveView's independently developed, open-source component library, and it has often driven LiveView framework development forward by introducing (and battle-testing) new concepts quickly.

The :if and :for special attributes provide syntactic sugar for the <%= if ... do %> and <%= for ... do %> EEx calls you would normally expect to write in your templates.

The :let attribute works a little bit differently. It is used in component slots when you want to yield a variable from within the function component back up to the caller. You'll see this most often when you use the .form/1 function component, but you can also use it to make your own function components even more dynamic.

Let's take a look at how you can use these attributes to write eloquent, ergonomic HTML in your LiveView templates. Along the way, you'll combine the usage of all three of these attributes to build a clean and dynamic LiveView function component.

Cleaner LiveView Templates with `:if` and `:for`

You can use these two attributes in regular Phoenix templates, in components, and in slots to write cleaner HTML code. We'll take a look at an example using the :if directive first:

<div class="alert alert-danger" :if={@error_message}>
  <p><%= @error_message %></p>
</div>

The div with the alert alert-danger class that contains the error message markup will only render if the @error_message is present and evaluates to true.

This approach replaces the more verbose one here:

<%= if @error_message do %>
<div class="alert alert-danger">
  <p><%= @error_message %></p>
</div>
<% end %>

The code that uses the :if attribute is easier to read--more eloquent--and easier to write--more ergonomic.

You can also use the :if attribute to conditionally render a function component, all in one simple line of code. Let's say we have a function component, .error/1, that wraps up the error message markup above. We can conditionally render it like this:

<.error message={@error_message} :if={@error_message}>

Now, let's take a look at an example that uses the :for attribute. The :for attribute can be used to iteratively render some content for each member in a collection, like this:

<tbody id="books">
  <tr :for={book <- @books}>
    <td><%= book.title %></td>
  </tr>
</tbody>

Thanks to the :for attribute, you don't have to establish your for loop explicitly within EEx tags, like this:

<tbody id="books">
  <%= for book <- @books %>
  <tr>
    <td><%= book.title %></td>
  </tr>
  <% end %>
</tbody>

Once again, we're left with code that is both more eloquent and more ergonomic. You can even combine the usage of :if and :for if you need to. Let's say we have a list of books to render, but we only want to display a given book if it is available in our online store. We can achieve that like this:

<ul>
  <li :for={book <- @books} :if={book.available}><%= book.title %></li>
</ul>

Since both :if and :for can be used in either plain HTML or in LiveView function components, we can wrap our list item HTML markup in a function component. Given the following function component:

def list_item(assigns) do
  ~H"""
  <li>
    <%= @book.title %>
  </li>
  """
end

We can call on it like this:

<ul>
  <.list_item :for={book <- @books} :if={book.available} />
</ul>

This is a nice way to start compartmentalizing our markup into reusable function components made even cleaner with special HTML attributes.

We can do better with our function component, though. Right now, we've only encapsulated the markup for list items, not the entire list. On top of that, our .list_item/1 function component isn't very reusable--it expects to be called with an assigns that contains an @book attribute set to something that responds to .title.

We'll build a more dynamic function component that uses component slots and the :let attribute to encapsulate the entire unordered list into one dynamic component. When we're done, you'll have a dynamic function component that uses :for and :let to render any collection, anywhere in your LiveView app.

For this next example, we'll simplify our logic a bit by dropping the :if attribute and the requirement that a book should only render if it's available. We'll just focus on using :let and :for. Let's get started.

Dynamic Function Components with `:let`

First up, let's use a named component slot to create a function component that renders an unordered list and its list items. Start by defining a top-level .unordered_list/1 function component that renders a named slot, :list_item, inside the appropriate markup:

def unordered_list(assigns) do
  ~H"""
  <ul>
    <li>
      <%= render_slot(@list_item, ...) %>
    </li>
  </ul>
  """
end

We're on track to render our unordered list component like this:

<.unordered_list>
  # ...
</unordered_list>

When we call on the component, we'll set an assigns - items - equal to the list of books. Now, we can use :for in the function component HEEx to render a list item slot for each item in the @items assigns:

def unordered_list(assigns) do
  ~H"""
  <ul :for={item <- @items}>
    <li>
      <%= render_slot(@list_item, item) %>
    </li>
  </ul>
  """
end

Putting it all together, we can call on our function component like this:

<.unordered_list items={@books}>
  <:list_item :let={book}>
    <%= book.title>
  </:list_item>
</.unordered_list>

Here's where the :let special attribute comes in. In our function component, we are using :for to iterate over the list of books in the @items assignment. For each book in the list, bound to the item variable at each step in the iteration, we are calling render_slot/2 with a second argument of item.

The call to :let={book} when we call on our :list_item slot sets a variable - book - equal to the second argument passed into render_slot/2. In this way, you yield variables from your function components back to the caller with the :let attribute. So, where we call :list_item when rendering the function component, we can operate on the book variable to display its title.

By combining :for and :let, we end up with an eloquent, ergonomic, and highly dynamic function component that we can use again and again in our application to render any collection into an unordered list.

Before we wrap up, there's one limitation that I'd like to point out here. Bringing back our "only render a book title if the book is available" logic is a little tricky here. You might want to do something like this:

<.unordered_list items={@books}>
  <:list_item :let={book} :if={book.available}>
    <%= book.title>
  </:list_item>
</.unordered_list>

This is not possible, however. The component slot call cannot combine :let with :if in that manner. We can't take advantage of the :if attribute here and instead have to write something like this:

def unordered_list(assigns) do
  ~H"""
  <ul :for={item <- @items}>
    <%= if item.available %>
      <li>
        <%= render_slot(@list_item, item) %>
      </li>
    <% end %>
  </ul>
  """
end

The downside is that the component becomes aware of the need to call .available on the item, making it less reusable.

Wrap Up

LiveView is being adopted so quickly in the Elixir community (and beyond) partly because of its gentle learning curve and emphasis on developer happiness.

LiveView's new special HTML attributes make writing LiveView templates even easier and more fun. By borrowing ergonomic features from Surface, we end up with code that is painless to write (fewer onerous EEx tags!) and easy to understand and maintain.

Reach for these special HTML attributes when writing plain HEEx templates, LiveView HEEx templates, or function components. You'll end up with beautiful code.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Faster XML Parsing with Elixir

Roy Tomeij — 2022-10-04T00:00:00+00:00

The XML data format has been around since 1996. It was first envisioned as a lingua franca (bridging language) for data to be serialized and read into completely disparate systems (with different programming languages, operating systems, and even hardware). It has been wildly successful in that goal.

In software, though, 26 years is like a lifetime — and in hardware, it's an eternity. So much has changed in that short time that it's easy to forget just how different the world was when the standard was proposed.

That brings us to the topic of today's post. How can we parse XML quickly and efficiently? What does state-of-the-art XML parsing look like?

Let's get going!

XML Parsing with Elixir — An Introduction

As much as we may want to criticize XML (and there is plenty to criticize), we have to put it in its historical context. Anything that survives that long has clearly found some utility!

Thinking critically about XML as a format is great if you are in a position to change or improve it, or when deciding whether you should use it.

But for most, the choice is already made for us. The entire airline industry and much of finance run on XML — so if you work in those industries, you will likely need to parse it.

We will define parsing in this post as doing two things:

Verifying a document is valid (i.e., that it is actually XML).
Extracting values from that valid document.

A valid document is not much use if we can't access the values in our program's XML. We will also specifically talk about XML documents where we know the expected shape of the XML ahead of time.

We'll provide an overview of some possibilities when parsing XML, including actual numbers from real-world applications.

Three Approaches to XML Parsing with Elixir

There are three broad approaches we will talk about:

Building a DOM
Virtual Token Descriptor (VTD)
Simple API for XML (SAX) parsing

Building a DOM

Building a DOM from XML is probably the most common way to think about XML parsing. It's what browsers do with HTML, and it preserves the tree structure that can make thinking about and validating an XML document much easier.

DOM stands for Document Object Model — but it really means a structured representation of XML in your given language. Elixir doesn't have Objects, but we can still have a DOM.

There are lots of mature tools in this area. Erlang even comes with a built-in data structure for representing an XML document called xmerl.

Xmerl represents XML so that XPath queries are possible, and an XPath query engine is also built into Erlang. What is XPath? It is to XML what SQL is to a database — a query language for accessing the data inside it.

For example, here is an XPath query to get the text from the XML snippet below: /Author/text()

<Author>James Eagan</Author>

You can use built-in functions to access the text in the tags:

xml = "<Author>James Eagan</Author>"
# First build the xmerl:
{xmerl, _} = xml |> :erlang.binary_to_list |> :xmerl_scan.string()

# Now we can apply the xpath.
:xmerl_xpath.string('/Author/text()', xmerl)

This tooling maturity can be a fast way to get up and running. There is a chance your team knows of XPath from other languages, which makes extracting data a breeze, and using the built-in xmerl format means 0 extra dependencies.

But there is a problem. Resources! With large XML files creating an xmerl, DOM can really balloon memory requirements. It can become prohibitively expensive in terms of latency and memory. Here is a Benchee output excerpt of a ~9mb XML file that uses xmerl and XPath:

Benchmarking xmerl with querying ...

Name                                   ips        average  deviation         median         99th %
...
xmerl with querying                 0.0809        12.36 s     ±1.98%        12.42 s        12.59 s

Comparison:
...
xmerl with querying                 0.0809 - 12.68x slower +11.39 s

Memory usage statistics:

Name                            Memory usage
...
xmerl with querying               2340.31 MB - 10.22x memory usage +2111.41 MB

That's a lot of megabytes! And as the output indicates, we can do much better.

Virtual Token Descriptor (VTD) in Elixir

VTD stands for Virtual Token Descriptor. A VTD is a completely different way to think about parsing XML. Instead of building a DOM, the idea is to tokenize the XML document and store the locations of key bits of information inside it.

In Elixir, this could mean storing the start index and length of all the nodes and attrs in the XML. You then use binary_part to access any given location (in constant time!) and extract values when needed.

We leave the original XML binary untouched in a VTD approach (unlike a DOM, where we manipulate the original XML binary). Instead, in a VTD we build up all the information we need to query the original binary.

VTD to XML Parsing: An Example

Let's imagine a simplified example. This is not how VTD works exactly, but it will give you a sense of what it does without getting bogged down in edge cases.

Given this XML:

<Author age="22">Seth Milchick</Author>

Let's imagine the information we want to be able to retrieve is:

Node name — "Author"
age attribute — "22"
<Author>'s text content

To do that, we could build a table that looks like this:

xml = "<Author age=\"22\">Seth Milchick</Author>"

table = [
  {:node, 1, 5},
  {:attribute, 8, 3},
  {:attribute_value, 13, 2},
  {:text, 17, 13},
]

The tuple denotes what we are pointing at, e.g., :node, then provides the start index for that element. The last number in the tuple is how long the element is.

If we look to get the XPath /Author/text(), we can access that table and see if the root node matches "Author", something like:

node_in_table? = Enum.any?(table, fn
  {:node, start, length} -> binary_part(xml, start, length) == "Author"
  _ -> false
end)

if node_in_table? do
  case Enum.find(table, &match?({:attribute_value, _, _}, &1)) do
    nil -> :not_text
    {:attribute_value, start, length} -> binary_part(xml, start, length)
  end
else
  :not_found
end

As you can see, how we represent that table is very important. This is the secret sauce of VTD. If we were inefficient, we could end up with a table the same size (or larger!) than a DOM. Or we could take a long time to find the element we care about in the table, eliminating any potential performance gains from binary_part.

An actual VTD table stores references to child and sibling nodes in a way that makes randomly accessing a given element as good as in a DOM. However, because we don't chop up the original binary into its own objects, the memory requirements can be vastly lower. The creators claim that the table's size can only be 1.5 times the size of the XML document. Lower memory requirements also mean it can be fast.

In fact, the parsing method was designed for on-chip implementations. This 'XML on a Chip?' document describes how you can use custom hardware to blaze through XML!

What's more, existing VTD implementations all support the full XPath feature set. That means it is (in theory) possible to have custom hardware churn through XML parsing without incurring a massive cost in developer complexity. I say 'in theory' because you first have to create the custom hardware, and then execute it with a suitably low-level language.

In fact, there are currently no Elixir libraries for VTD. I know of two libraries being worked on (one in Erlang and one in Elixir) as part-time open source projects, as of yet unfinished and unreleased. Perhaps you can take inspiration from other languages and have a go at porting it.

As cool and interesting as this approach sounds, though, there are more trade-offs.

Trade-offs of VTD

While XPath is probably familiar to some, it...isn't great. When a node in a path doesn't exist, you just get nil or an empty string back. This is a pain because you are never sure if you mistyped the path or if the value actually isn't there. Verifying that can be manual and annoying. XPath functions also don't tell you which node doesn't exist, making debugging harder. Having XPath as the primary way you extract data from an XML document can prove quite painful.

Additionally, VTD requires an entire document to exist in memory first, unlike sax parsing (as we will see). This could make parsing a stream of XML with VTD difficult.

Finally, there is a minimum amount of data that we need to extract from an XML document. We need to turn that data into useful things (like an Elixir Date struct) and those Elixir data structures are of a certain size. If they amount to more in memory than the parsing takes, we don't gain anything from further reducing the memory footprint of the parsing. In some situations, the impressive memory savings don't translate to an actual saving in our programs (because as soon as you transform the parsed data into structured data, the memory jumps up again anyway).

So, let's look at our final approach — sax parsing.

SAX Parsing in Elixir

A SAX (Simple API for XML) parser works by iterating through a document and emitting events when certain things happen. For example, the Saxy parser emits the following events.

:start_document
:start_element
:characters
:cdata
:end_element
:end_document

When Saxy sees the start of a tag, for example, it gets its name and attributes and calls a callback you can implement. You can do whatever you like with that information.

The specific events available vary by implementation. Erlang has a built-in SAX xmerl parser that offers extra events like :comment and :ignorableWhitespace.

The emission of events is usually very fast. In Elixir, we can use binary pattern matching to great effect; Saxy will iterate through a document and use pattern matching to extract node names and attributes. But this only achieves half of what you actually need. You then have to identify when a value is one you care about, which is not as simple as it sounds.

SAX Parsing: An Example

Let's imagine you want to access the text inside the <Name> tag below:

<Author>
  <Name>Nick Riviera</Name>
</Author>

For this simple case, you first need to wait for the :start_element event where the node_name is Name:

def handle_event(:start_element, {"Name", attrs}, state) do
  # In here we know we have just opened the Name tag.
end

# This case is for any other tag that we open.
def handle_event(:start_element, {_, _attrs}, state) do
 {:ok, state}
end

But the :characters event is separate, meaning we have to put something into the state to let us know which tag's characters we're seeing:

def handle_event(:start_element, {"Name", attrs}, state) do
  # In here we know we have just opened the Name tag, so let's just put that in state.
  {:ok, [{"Name", attrs} | state]}
end

def handle_event(:start_element, {_, _attrs}, state) do
 {:ok, state}
end

Now, when we get the :characters event, we can check the state and see if we are "inside" the tag we care about:

# If "state" is the name tag we know it's the text we care about.
def handle_event(:characters, text, [{"Name", _} | rest]) do
  # Now we can do something with the text since we know it will be the text of <Name>.
  # ...
end

def handle_event(:characters, _, state) do
 {:ok, state}
end

That works great, but now consider this XML:

<Blog>
  <Name>XML Parsing</Name>
  <Author>
    <Name>Nick Riviera</Name>
  </Author>
</Blog>

Our approach will break here because we need to know which <Name> node we have encountered.

The solution is to use a stack. Each time we open a tag, we add that element to the stack, and when we close an element, we can discard it.

Trade-offs of a Stack

Changing our mindset to think in stacks is not simple, especially when we normally think in recursion.

It also makes validating the XML more complicated. If you don't get the value that you expect, it can be difficult to figure out what went wrong and when. You can print the stack, but figuring out how it got to be like that basically requires implementing your own stack tracing algorithm. So debugging is not trivial.

Implementing the callbacks for each XML document you want to parse can be an awful lot of work. It is difficult to re-use shared code across handlers because each time you accumulate different state. You might want to create an %Author{} struct for this:

<Author>
  <Name>Nick Riviera</Name>
</Author>

But for this document:

<Flight departureTime="1pm" />

You might want a %Flight{} struct.

The only generic functionality is to manage the stack as you progress through a document. However, the specific elements and what you want to pull out of a document and put into the state you accumulate will be different. You really have only two options:

Write a unique handler per document that accumulates all the data you want to pull from the XML document.
Have a generic handler used by every document, which puts everything into a generic XML data structure.

Number 1 allows you to collect only the data you wish to pull from the XML and ignore the rest but it requires writing a custom handler for each XML document.

And number 2 — well, number 2 is a DOM 😅.

By default, Saxy actually does number 2 — it creates a data structure called SimpleForm, a tuple DOM that looks like this:

{node_name, attributes, children}

Where children is a list of more tuple nodes or text — for example:

<Blog>
  <Name>XML Parsing</Name>
  <Author>
    <Name>Nick Riviera</Name>
  </Author>
</Blog>

Becomes:

{"Blog", [], [
  {"Name", [], ["XML Parsing"]},
  {"Author", [], [
    {"Name", [], ["Nick Riviera"]}
  ]}
]}

Great...but haven't we just run slap-bang into all of the DOM problems we mentioned above? Well, almost. We are in full control of the created DOM, which means we can be clever.

The xmerl DOM we mentioned before is designed to enable XPath, but if we can live without that (and I maintain that we can, nay, we can improve on it), we can significantly reduce the size of the DOM we create.

We need to be willing to do two things:

Replace XPath with something.
Design a more minimal DOM.

I have seen amazing improvements over xmerl doing just this; an order of magnitude less memory and a latency improvement of the same.

For 2, we can go with SimpleForm for now, but let's look at what we can do about 1.

Querying the DOM with DataSchema

While a custom query engine may sound scary, it's an opportunity to improve. Opting to navigate a DOM in Elixir means we don't have a black box XPath implementation that just returns nil when it's not happy (leaving you to figure out if the data wasn't there or if you just typed the path wrong).

Usually, everything we ever need from the XML boils down to one of three things:

A node
A node's text
A node's attr

We either want one, or many, of each of these things. We end up with an API that's something like:

get_text
get_attr
get_all_attrs
get_all_text
get_node
get_all_nodes

How DataSchema Works

To help improve the developer experience even further, I wrote DataSchema. DataSchema is like Ecto's embedded schemas but generalized to any input type, including XML.

It allows us to write declarative schemas that describe structs we can create from input data. For example, this is a schema for the XML snippet above:

<Blog>
  <Name>XML Parsing</Name>
  <Author>
    <Name>Nick Riviera</Name>
  </Author>
</Blog>

defmodule Blog do
  import DataSchema, only: [data_schema: 1]

  @data_accessor SimpleFormAccessor
  data_schema(
    field: {:title, ["Blog", "Name", "text()"], StringType},
    field: {:author_name, ["Blog", "Author", "Name", "text()"], StringType},
  )
end

There are some really good livebooks in the repo to learn the library properly, but let's break down the basics of the schema.

Each field in a schema defines a key in a struct. You can see the key below:

field: {:title, ["Blog","Name","text()"], StringType},
 #      ^^^^^^^
 #  The struct key

The next element in that tuple is a path to a piece of data in the XML:

field: {:title, ["Blog","Name","text()"], StringType},
 #                   ^^^^^^^
 #               Path to a value

We are in full control of what that path can look like, but for now, I've opted to make it a list of XML nodes that mirror what an XPath query would look like split on "/".

The last element in that tuple is a casting function. When we call:

DataSchema.to_struct(xml, Blog)

DataSchema will get the value at the end of the path (for each field) and pass it to a casting function. This gives us a chance to alter it in some way.

field: {:title, ["Blog","Name","text()"], StringType},
 #                                         ^^^^^^^
 #                                    casting function

The casting function can be a module that implements a cast/1 function, an anonymous fn, or a Mod Fun Args tuple.

Finally, DataSchema needs to apply the path to the input data it transforms.

We provide a @data_accessor in the schema:

...
@data_accessor SimpleFormAccessor
#                 ^^^^^^^^^
#              Accessor module
...

This module will implement a callback that gets passed the path and the input data (in our case, XML). The return value of that function is given to the casting function.

This accessor is where we can implement the traversal of the SimpleForm data structure. This is relatively easy. Here is an example snippet getting the text of an element:

def get_text(["text()"], {_, _, children}) do
  Enum.filter(children, &is_binary/1)
end

def get_text([node_name | rest], {node_name, _, children}) do
  get_text(rest, children)
end

def get_text([node_name | _], _) do
  {:error, "node not found #{node_name}"}
end

This small snippet demonstrates the improvement we can get over XPath because we can now log out the exact node that we could not find. This is a huge developer experience productivity boost.

All this comes together so that when we call DataSchema.to_struct(xml, Blog), it returns:

{:ok, %Blog{title: "XML Parsing", author_name: "Nick Riviera" }}

This is the flow of data:

XML binary =>
  Saxy (to create a SimpleForm DOM) =>
    Query that DOM with DataSchema =>
      Struct

XML Parsing with Elixir: Taking it One Step Further

In the wild, I've found that the Saxy/DataSchema approach improves over the xmerl DOM by order of magnitude. But we can go one step further.

Our schemas are declarative specifications of all the data we want from the XML. We can feed this information into a saxy handler and only include elements in the DOM that our schema needs.

This nifty trick works out especially well for schemas where you only need a small amount of the data in the XML document. If the data you end up with is larger in memory than the DOM you build, you don't need to worry about making the DOM smaller. But if it isn't, then building the DOM can cause waste and garbage, as you end up ignoring most of it.

We need to transform our schema into a tree with the schema's paths — e.g., for this Blog schema:

defmodule Author do
  import DataSchema, only: [data_schema: 1]

  @data_accessor SimpleFormAccessor
  data_schema(
    field: {:name, ["Author", "text()"], StringType},
  )
end

defmodule Blog do
  import DataSchema, only: [data_schema: 1]

  @data_accessor SimpleFormAccessor
  data_schema(
    field: {:title, ["Blog", "Name", "text()"], StringType},
    has_one: {:author, ["Blog", "Author"], StringType},
  )
end

We would create a tree of paths that looks something like:

%{
  "Blog" => %{
    "text()" => true,
    "Author" => %{
      "text()" => true
    }
  }
}

Handing that tree to the Saxy handler allows us to skip entire sub-trees of the XML.

To demonstrate this, let's parse the XML with the above schema.

<Blog>
  <Comments>
    <Comment>Superb!</Comment>
    <Comment>Amazing!</Comment>
  </Comments>
  <Name>XML Parsing</Name>
  <Author>
    <Name>Nick Riviera</Name>
  </Author>
</Blog>

First, recall that the key to parsing in this manner is to create a stack of elements. As you can see, when we build a SimpleForm DOM, we maintain a stack of nodes. When we get a :start_element event, we put that element onto the top of the stack.

When we receive an :end_element event, we put the element that is on the top of the stack into the children of the element below it (its parent).

But this time our handler has a schema and therefore a record of the values we actually care about. We can now query it every time we start a new element and ask "is this element in the schema?". If it is, we do as explained above and add it to the stack. If it isn't, we add a special node to the stack:

{:skip, 1, tag_name}

When we close a tag that we are skipping (i.e. when the top of the stack is that tuple) then we just pop that {:skip, 1, tag_name} element off of the stack and discard it. But the real win is if we open another tag before that one closes — i.e. when we meet the child of a skipped element.

In the example above, it would go like this:

We open <Comments> and see it is not in the schema, so we add :skip.
We get an event for the start of the <Comment> tag. We see that the top of the stack is a :skip element, so we immediately return:

def handle_event(:start_element, _element, [{:skip, _, _} | _] = stack) do
  {:ok, state}
end

This continues until we close <Comments> and the :skip element is removed. Then we continue as normal.

Okay, but what is that 1 doing in the middle? This is effectively a reference count. Consider this XML:

<Blog>
  <Comments>
    <Comments>
    </Comments>
  </Comments>
</Blog>

While nasty, it's perfectly valid XML that risks the following:

We add a :skip tag for the opening of the first <Comments>.
The opening of the second <Comments> is skipped.
On the closing of the second <Comments>, the name of the "skipped" tag matches this one, so we remove the skipped element from the stack, even though we haven't actually finished skipping yet.

One solution to this is to add a :skip tag for every element you want to skip. But then you don't save any memory, as you still end up with a stack of every element — you just remove it more quickly. The memory will seem lower but will spike high during parsing.

Instead, we'll keep a count of elements with the same name as the skipped element once we start skipping. Then, each time we close one of those elements, we decrement the count. If it's ever 0, we are done skipping and can remove the tag.

This approach speeds up the latency of parsing a little, but importantly (along with some other tricks) massively trims down the size of the SimpleForm DOM that gets created.

See a version of the full Saxy handler.

There is even more we could do in this vein. For example, this approach stores all nodes along the path to the value we want. But if we just want the value at the end of the path, could we keep that value and none of the intermediary nodes? This comes with its own problems. We'll leave discovering these problems as an exercise for you 🙂.

Wrap Up

This concludes our tour of some interesting ways to parse XML faster, including building a DOM, using a VTD, and SAX parsing. We also went one step further, feeding data into a Saxy handler (including elements in the DOM needed by our schema).

My hope is to one day open source a complete implementation of the slimmed down Sax parser, but for now, I hope this post has stimulated some of your own ideas.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Fix Process Bottlenecks with Elixir 1.14's Partition Supervisor

Roy Tomeij — 2022-09-20T00:00:00+00:00

Elixir v1.14 shipped earlier this month with a bunch of new goodies.

In this post, we'll explore Elixir's new PartitionSupervisor. We'll take a look at some code that suffers from the exact bottleneck issue that partitions supervisors are designed to solve. Then, we'll fix that bottleneck. Along the way, you'll learn how partition supervisors work under the hood to prevent process bottlenecks.

Let's get started!

The Problem: Dynamic Supervisor Bottlenecks in Your Elixir App

You may be familiar with using dynamic supervisors to start child processes on-demand in your application. Telling the dynamic supervisor to start its children can become a bottleneck, however.

With just one dynamic supervisor, concurrent processes each have to wait their turn to tell your app to start up children. If the "start up child" process takes a long time, or your app handles a very high volume of "tell the dynamic supervisor to start children" requests, you've got a bottleneck. The single dynamic supervisor can only initialize one child process at a time.

Let's take a look at a simple example that artificially replicates such a bottleneck.

The Slow Worker and Its Dynamic Supervisor

In my sample app here, I have a worker - SlowGreeter.Worker. The worker module is a genserver that initializes by sleeping for five seconds and then printing out a greeting:

defmodule SlowGreeter.Worker do
  use GenServer

  def start_link(%{name: name, from: from}) do
    GenServer.start_link(__MODULE__, %{name: name, from: from})
  end

  def init(state) do
    :timer.sleep(5000)
    send(self(), :greet)
    {:ok, state}
  end

  def handle_info(:greet, %{name: name, from: from}) do
    IO.puts("Hello from #{from}, #{name}")
    {:noreply, name}
  end
end

Disclaimer: This is meant to replicate a genserver with some time-consuming work to do on initialization. It should be noted that a "slow-starting" genserver is something of a code smell. You should avoid doing time-consuming work in the #init/1 function and leverage handle_continue/2 to prevent long-running tasks from blocking genserver start up.

But, sometimes doing such work when the genserver starts up is unavoidable. And regardless, this setup helps us replicate a bottleneck scenario that can happen:

1. When your dynamic supervisor is either starting up a slow-to-initialize worker, or

2. If your app is handles many concurrent requests telling the same dynamic supervisor to start up its children.

Okay, with that disclaimer out of the way, let's proceed with illustrating our bottleneck. We have our slow-to-initialize worker, and we also have a SlowGreeter.DynamicSupervisor module that starts it up:

defmodule SlowGreeter.DynamicSupervisor do
  use DynamicSupervisor
  alias SlowGreeter.Worker

  def start_link(init_arg) do
    DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__)
  end

  @impl true
  def init(_init_arg) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  def start_greeter(name) do
    spec = %{id: Worker, start: {Worker, :start_link, [%{name: name, from: "DynamicSupervisor"}]}}
    DynamicSupervisor.start_child(__MODULE__, spec)
  end
end

The SlowGreeter.DynamicSupervisor starts when the application starts up. Then, we can tell it to start the greeter worker like this:

SlowGreeter.DynamicSupervisor.start_greeter("Frankenstein")
# sleep for 5 seconds...
"Hello from DynamicSupervisor, Frankenstein"

Now we're ready to demonstrate our bottleneck.

Demo: The Bottleneck

Let's spawn five processes, each of which will make a call to SlowGreeter.DynamicSupervisor.start_greeter/1. Since each of our spawned processes requests the same dynamic supervisor, we'll see that each process must wait its turn.

for _i <- 1..5 do
  name = Faker.Person.first_name()
  # The DynamicSupervisor starts a worker that sleeps for 5 seconds then puts a greeting
  spawn(fn -> SlowGreeter.DynamicSupervisor.start_greeter(name) end)
end

When the SlowGreeter.DynamicSupervisor starts up its child SlowGreeter.Worker, it must wait for the worker to initialize. The worker initialization process is slow--we told it to sleep for five seconds and then print out a greeting. We'll see that each spawned process's call to the dynamic supervisor is only processed when the preceding one finishes. As a result, each greeting prints out at a five-second interval, as you can see in this video:

This approach is bottlenecked by the dynamic supervisor itself. It can only process one "start up child" request at a time, and must wait until the initialization of a child worker is complete before moving on to the next one.

We can remove this bottleneck with the help of Elixir 1.14's new PartitionSupervisor. Let's implement it now. Then, we'll take a deeper dive into how it works.

The Fix: Supervise a Fleet of Dynamic Supervisors with a Partition Supervisor

Instead of initializing just one dynamic supervisor when our application starts up, we'll use the new PartitionSupervisor to spin up a set of dynamic supervisors.

Then, instead of making direct calls to the dynamic supervisor to start its child worker, we will go through the partition supervisor, which will route the request to one of the many dynamic supervisors it's overseeing. Let's take a look at the code.

Using `PartitionSupervisor` in the Dynamic Supervisor Module

We'll implement a new module, SlowGreeter.DynamicSupervisorWithPartition, which uses the DynamicSupervisor behaviour. This time, however, the #start_greeter/1 function will call on the dynamic supervisor to start its child via the PartitionSupervisor.

defmodule SlowGreeter.DynamicSupervisorWithPartition do
  use DynamicSupervisor
  alias SlowGreeter.Worker

  def start_link(init_arg) do
    DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__)
  end

  @impl true
  def init(_init_arg) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  def start_greeter(name) do
    spec = %{id: Worker, start: {Worker, :start_link, [%{name: name, from: "PartitionSupervisor"}]}}

    DynamicSupervisor.start_child(
      {:via, PartitionSupervisor, {__MODULE__, self()}},
      spec
    )
  end
end

Here's where the magic happens:

DynamicSupervisor.start_child(
  {:via, PartitionSupervisor, {__MODULE__, self()}},
  spec
)

Instead of calling on the SlowGreeter.DynamicSupervisorWithPartition directly, we call it through the partition supervisor using the {:via, PartitionSupervisor, {dynamic_supervisor_name, routing_key}} tuple.

This will route the "start child request" to one of the running SlowGreeter.DynamicSupervisorWithPartition processes supervised by the partition supervisor that starts up with our application. Let's configure our app to start such a supervisor now:

# application.ex
def start(_type, _args) do
  children = [
      {PartitionSupervisor, child_spec: DynamicSupervisor, name: SlowGreeter.DynamicSupervisorWithPartition},
    ]

    opts = [strategy: :one_for_one, name: SlowGreeter.Supervisor]
    Supervisor.start_link(children, opts)
end

With this, when the app starts, a partition supervisor will start up and begin supervising a set of SlowGreeter.DynamicSupervisorWithPartition dynamic supervisors. The default behavior is to create a partition for each available scheduler (usually one per core of your machine). It will then place a dynamic supervisor on each partition.

With our code in place, let's see a demo of our bottleneck fix.

Demo: No More Bottleneck

Once again, we'll spawn five processes. Each process will call on the SlowGreeter.DynamicSupervisorWithPartition#start_greeter/1 function, like this:

for _i <- 1..5 do
  name = Faker.Person.first_name()
  # The PartitionSupervisor tells one of its dynamic supervisors to start a worker that sleeps for 5 seconds then puts a greeting
  spawn(fn -> SlowGreeter.DynamicSupervisorWithPartition.start_greeter(name) end)
end

This function goes through the partition supervisor to route a "start child worker" request to dynamic supervisors across partitions. This means we don't have to wait for a single dynamic supervisor to finish initializing a child worker before moving on to process the next request from a spawned process.

Instead, the partition supervisor will route these requests so that they're processed more or less concurrently by the dynamic supervisors spread across partitions. So, we'll see that all five greetings are printed out simultaneously after just one five-second sleep. You can see this behavior in the video below:

With that, we've fixed our bottleneck! Remember that our worker's "sleep for five seconds on init" behavior is artificial, and you'll want to avoid lengthy worker startups in general. But, this example serves to illustrate the kind of bottleneck that can occur in your application if you deal with lots of concurrent requests to the same dynamic supervisor.

Elixir 1.14's new partition supervisor neatly solves this problem by partitioning your dynamic supervisors and distributing requests to them, so that they can handle a greater load.

Where you don't need to share state between child processes supervised by dynamic supervisors, this approach can help your dynamic supervisors handle scale and avoid bottlenecks.

Now that we've seen partition supervisors in action, let's take a closer look at how they work.

`PartitionSupervisor` in Elixir 1.14: Under the Hood

Keep reading for a deeper dive into how partition supervisors establish partitions and route requests to dynamic supervisors.

Partitioning Dynamic Supervisors

As we saw earlier, you can tell your app to start a partition supervisor, overseeing some dynamic supervisors, in your application.ex's #start/2 function like this:

def start(_type, _args) do
  children = [
    {PartitionSupervisor, child_spec: DynamicSupervisor, name: SlowGreeter.DynamicSupervisorWithPartition},
  ]

  opts = [strategy: :one_for_one, name: SlowGreeter.Supervisor]
  Supervisor.start_link(children, opts)
end

Here, we start a partition supervisor that starts a SlowGreeter.DynamicSupervisorWithPartition for each core in our machine. This is the default start_link behavior.

PartitionSupervisor#start_link/1 supports options that include a :partitions key. The :partitions key should be set to a positive integer representing the number of partitions and defaults to System.schedulers_online(), which should return the number of cores on your machine.

Under the hood, a partition supervisor is just a regular supervisor that generates a child spec for each partition and then stores the partition info in ETS or a Registry. Later, when you tell a dynamic supervisor to start its child via the partition supervisor, it selects a partition based on the routing algorithm and routes the call to a dynamic supervisor in that partition. So, it effectively spins up a set of dynamic supervisors when your app comes up, and then starts child processes across those dynamic supervisors.

Let's take a closer look at the routing behavior now.

Routing Requests to Dynamic Supervisors

When you're ready to start a dynamic supervisor's child through a partition supervisor, you do so like this:

DynamicSupervisor.start_child(
  {:via, PartitionSupervisor, {dynamic_supervisor_name, routing_key}},
  spec
)

The :via tuple specifies the name of the dynamic supervisor to start up and a routing key.

In our example app, we used self() as a routing key, which will return the PID of the current process, but you can also provide a positive integer here. If key is an integer, it is routed using rem(abs(key), num_partitions). Otherwise, the routing is performed by calling :erlang.phash2(key, num_partitions). So, when you use the same PID as a routing key, you can be assured that your call will be directed to the same partition every time.

Wrap Up

We've seen how you can use partition supervisors to avoid bottlenecks in your dynamic supervisor code. Provided the state of your child processes can be partitioned (i.e., you don't need to share state among them), you can reach for a partition supervisor to elegantly scale up your dynamic supervisors.

In fact, partition supervisors can be used to partition and scale any process in your Elixir application. If a process can be started up, and have its state partitioned, then you can use a partition supervisor. Start the partition supervisor with its child spec (just like we did for our dynamic supervisor) and dispatch messages to the partition supervisor's children using the same :via tuple we demonstrated.

This is just one of the many exciting new features of Elixir v1.14. Read about another new feature, debugging with dbg()!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Elixir 1.14: Better Debugging with dbg/2 and More

Roy Tomeij — 2022-09-13T00:00:00+00:00

The latest Elixir release introduces new features to improve your developer and debugging experience.

In this post, we'll take a look at the new dbg() functionality, along with some improvements to Inspect and binary evaluation error messaging. All these changes come together to make you an even more productive Elixirist.

Let's get started!

Debugging with `Kernel.dbg/2` in Elixir

The Elixir Kernel now exposes a dbg/2 macro that you can use with or without IEx for enhanced debugging. Let's walk through a few examples that illustrate just how useful this tool can be in your debugging toolkit.

In this example, we have a Phoenix LiveView application called Live Library. Users can browse, checkout, and return books. Unfortunately for us, however, it looks like we have a bug--when a user tries to check out a book, the live view crashes:

We can use the dbg/2 function to gain insight into what's going wrong. We'll start with a few well-placed calls to dbg/2. Then, we'll move on to leveraging dbg/2 within an IEx session.

Debugging the Current Binding

First, we'll make a call to dbg/2 in the live view's event handler function that gets called when the user clicks "checkout". This is the entry point of our "checkout book" code flow, so we'll start our debugging process there.

# lib/live_library_web/live/book_live/checkout_component.ex

def handle_event("checkout", _, %{assigns: %{book: book}} = socket) do
  dbg() # this is new!
  case Library.checkout_book(book, socket.assigns.current_user.id) do
    {:ok, book} ->
      {:noreply, socket |> assign(:book, book)}

    {:err, _err} ->
      {:noreply, socket}
  end
end

Now, if we try once again to check out a book, we'll see the following output in our terminal where we're running the Phoenix server via mix phx.server:

[(live_library 0.1.0) lib/live_library_web/live/book_live/checkout_component.ex:33: LiveLibraryWeb.CheckoutComponent.handle_event/3]
binding() #=> [
  book: %LiveLibrary.Library.Book{
    updated_at: ~N[2022-09-11 17:42:19],
    inserted_at: ~N[2022-02-28 16:33:45],
    book_loans: [
      %LiveLibrary.Library.BookLoan{
        updated_at: ~N[2022-09-11 17:39:18],
        inserted_at: ~N[2022-09-11 17:39:18],
        book_id: 7,
        user_id: 4,
        id: 51,
        __meta__: #Ecto.Schema.Metadata<:loaded, "book_loans">
      }
    ],
    author_id: 4,
    title: "My Test Book!!!",
    description: "A book I made up. Different from the book I wrote which I also technically made up.",
    cover: "live_view_upload-1657544569-150520820351856-7",
    available: true,
    id: 7,
    __meta__: #Ecto.Schema.Metadata<:loaded, "books">
  },
  socket: #Phoenix.LiveView.Socket<
    id: "phx-FxPfPGQDjqGwUAyB",
    endpoint: LiveLibraryWeb.Endpoint,
    view: LiveLibraryWeb.BookLive.Index,
    parent_pid: nil,
    root_pid: #PID<0.780.0>,
    router: LiveLibraryWeb.Router,
    assigns: %{
      __changed__: %{},
      book: %LiveLibrary.Library.Book{
        updated_at: ~N[2022-09-11 17:42:19],
        inserted_at: ~N[2022-02-28 16:33:45],
        # ...
      },
      current_user: #LiveLibrary.Accounts.User<
        updated_at: ~N[2022-02-18 14:49:02],
        inserted_at: ~N[2022-02-18 14:49:02],
        admin: true,
        confirmed_at: nil,
        email: "admin@email.com",
        id: 4,
        __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
        ...
      >,
      flash: %{},
      id: "checkout-7",
      myself: %Phoenix.LiveComponent.CID{cid: 5}
    },
    transport_pid: #PID<0.768.0>,
    # ...
  >
]

When called without any arguments, dbg/2 defaults to debugging the binding for the current context. In this case, that's the handle_event/3 function (and the variables available to us by the time the dbg/2 call is hit on the first line of that function).

As you can see in the output above, calling dbg/2 does a few things for us - printing out the:

Location of the debugged code:

[(live_library 0.1.0) lib/live_library_web/live/book_live/checkout_component.ex:33: LiveLibraryWeb.CheckoutComponent.handle_event/3]

Value of the current context's binding:

binding() #=> [
  book: %LiveLibrary.Library.Book{
    #...

The call to dbg/2 also returns the value of the debugged code itself. We'll see how helpful this is in a bit.

For now, we're already experiencing some of the benefits of dbg/2 with just this basic usage. Let's examine the value of the arguments that handle_event/3 is called with. This way, we'll better understand how our code is executing and what might be going wrong.

Based on the dbg/2 output, it looks like the input to handle_event/3 is exactly what we expect. We have a socket whose assigns correctly contain a book and a current user. So, let's move our call to dbg/2 downstream a bit, into the Library.checkout_book/2 context function.

Debugging Provided Code

At this point in our debugging journey, we're ready to turn our attention to the behavior of the Library.checkout_book/2 function. This is the next step in our broken "checkout book" code flow.

We're interested in a few things. We'd like to understand what exact arguments the function is being called with, and what it returns. We can easily reveal this information with two calls to debug/2.

We'll call dbg/2 with the current default context at the beginning of the checkout_book/2 function to look at the arguments' values.

Then, we'll pipe the code in Library.checkout_book/2 into a call to dbg/2 at the end of the function. This will allow us to see the return value of the function. And, since dbg/2 returns the value of the executed code you pass to it as a first argument, the remainder of our "checkout book" code flow will be unaffected by this call to dbg/2.

Here's what our debugged function looks like:

def checkout_book(book, user_id) do
  dbg()
  with {:ok, _loan} <- Library.create_book_loan(%{user_id: user_id, book_id: book.id}),
       {:ok, book} <- Library.update_book(book, %{available: false}) do
        book
  else
    err -> err
  end
  |> dbg()
end

Now if we try to check out a book, we'll see the following output in our terminal:

[(live_library 0.1.0) lib/live_library/library.ex:13: LiveLibrary.Library.checkout_book/2]
binding() #=> [
  book: %LiveLibrary.Library.Book{
    updated_at: ~N[2022-09-11 18:27:59],
    inserted_at: ~N[2022-02-28 16:33:45],
    author_id: 4,
    title: "My Test Book!!!",
    description: "A book I made up. Different from the book I wrote which I also technically made up.",
    cover: "live_view_upload-1657544569-150520820351856-7",
    available: true,
    id: 7,
    __meta__: #Ecto.Schema.Metadata<:loaded, "books">,
    # ...
  },
  user_id: 4
]

# ...

[(live_library 0.1.0) lib/live_library/library.ex:21: LiveLibrary.Library.checkout_book/2]
with {:ok, _loan} <- Library.create_book_loan(%{user_id: user_id, book_id: book.id}),
     {:ok, book} <- Library.update_book(book, %{available: false}) do
  book
else
  err -> err
end #=>
%LiveLibrary.Library.Book{
  updated_at: ~N[2022-09-11 18:28:00],
  inserted_at: ~N[2022-02-28 16:33:45],
  # ...
}

From this, we can clearly see that the checkout_book/2 function returns a single book struct. We're close to solving our bug now.

Let's return to the handle_event/3 function that calls checkout_book/2. This time, we'll run our server in an IEx session with iex -S mix phx.server. Then, we'll place another dbg/2 call in the handle_event/3 function to freeze code execution and interact with our code in real time.

Debugging with `dbg/2` and IEx in Elixir

The dbg/2 function is configurable, and you can extend it with advanced behavior. IEx comes with one such extension baked in.

In Elixir 1.14, IEx extends dbg/2 with an interactive shell, similar to IEx.pry.

Let's add back our dbg/2 call to the event handler function and play around:

# lib/live_library_web/live/book_live/checkout_component.ex

def handle_event("checkout", _, %{assigns: %{book: book}} = socket) do
  dbg()
  case Library.checkout_book(book, socket.assigns.current_user.id) do
    {:ok, book} ->
      {:noreply, socket |> assign(:book, book)}

    {:err, _err} ->
      {:noreply, socket}
  end
end

Now when we try to check out a book in the browser, we'll see that we have a request to pry in our terminal:

Request to pry #PID<0.816.0> at LiveLibraryWeb.CheckoutComponent.handle_event/3 (lib/live_library_web/live/book_live/checkout_component.ex:33)

   32:   def handle_event("checkout", _, %{assigns: %{book: book}} = socket) do
   33:     dbg()
   34:     case Library.checkout_book(book, socket.assigns.current_user.id) do
   35:       {:ok, book} ->
   36:         Endpoint.broadcast(@checkout_topic, "checkout_event", %{book: book})

Allow? [Yn] Y

We can now interact with and execute our code. Let's manually execute the checkout_book/2 function like this:

pry(1)> Library.checkout_book(book, socket.assigns.current_user.id)
iex(1)> [debug] QUERY OK db=1.0ms idle=1240.2ms
INSERT INTO "book_loans" ("book_id","user_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4) RETURNING "id" [7, 4, ~N[2022-09-11 18:46:06], ~N[2022-09-11 18:46:06]]
[debug] QUERY OK db=1.0ms queue=0.3ms idle=1241.5ms
UPDATE "books" SET "available" = $1, "updated_at" = $2 WHERE "id" = $3 [false, ~N[2022-09-11 18:46:06], 7]
%LiveLibrary.Library.Book{
  updated_at: ~N[2022-09-11 18:46:06],
  inserted_at: ~N[2022-02-28 16:33:45],
  # ...
  ],

Here, we can see that we are returning the newly checked-out book. A call to whereami will show us the next bit of code flow that will try to execute:

pry(2)> whereami
Location: lib/live_library_web/live/book_live/checkout_component.ex:33

   31:
   32:   def handle_event("checkout", _, %{assigns: %{book: book}} = socket) do
   33:     dbg()
   34:     case Library.checkout_book(book, socket.assigns.current_user.id) do
   35:       {:ok, book} ->

    (live_library 0.1.0) lib/live_library_web/live/book_live/checkout_component.ex:33: LiveLibraryWeb.CheckoutComponent.handle_event/3

This reminds us that the case statement expects to match on a return value not of a single book struct, but rather of an {:ok, book} tuple. With that, we've found our bug! By updating the checkout_book/2 function to return a tuple instead of a book struct, we'll fix our problem.

Before we move on to some other new Elixir 1.14 features that will improve your development and debugging experience, let's take a last look at some additional dbg/2 functionality.

More `dbg/2` Goodies

Let's examine how dbg/2 makes it easier than ever before to debug pipelines. Then, we'll peek at some of the more advanced dbg/2 configuration capabilities.

Debugging Pipelines with `dbg/2`

The dbg/2 macro makes it easier to debug pipelines. Before dbg/2, if you wanted to examine return values at each step of your pipeline, you'd have to add an explicit call to IO.inspect at each step - like this:

# lib/live_library_web/live/book_live/index.ex
def mount(_params, %{"user_token" => user_token}, socket) do
  {:ok,
   socket
   |> assign_user(user_token)
   |> IO.inspect()
   |> assign_books()
   |> IO.inspect()
   |> assign_search()
   |> IO.inspect()
   |> assign_search_changeset()
   |> IO.inspect()
   |> assign_categories()
   |> IO.inspect()
   |> assign_category_ids()}
end

This isn't very eloquent and it can be tedious to implement. Now in Elixir 1.14, we can place just one call to dgb/2 at the end of our pipeline. This will print out the value at each individual step of the pipeline and ensure that the pipeline still returns the value of the executed code. Let's add in our dbg/2 call here:

def mount(_params, %{"user_token" => user_token}, socket) do
  {:ok,
   socket
   |> assign_user(user_token) # %{assigns: %{user: #...}}
   |> assign_books() # %{assigns: %{books: #...}}
   |> assign_search() # %{assigns: %{search: #...}}
   |> assign_search_changeset() # # %{assigns: %{search_changeset: #...}}
   |> assign_categories() # %{assigns: %{categories: #...}}
   |> assign_category_ids() # %{assigns: %{category_ids: #...}}
   |> dbg()}
end

Now, when we visit the /books index page that brings up this live view, we'll see that the return value of each step of the pipeline has been printed for us to examine.

Additionally, if you run this code in an IEx session, you will be able to step, or pry, through each individual step in the pipeline. Code execution will freeze at each successive line in the pipeline.

Pipelines represent one of Elixir's most powerful and eloquent language features, and debugging them just got easier.

Configuring `dbg/2`

You can extend the behavior of dbg/2 in your Elixir application by defining a custom function. Tell dbg/2 to use that custom function via your app's :dbg_callback configuration key. Your app configuration should look like this:

# config/config.exs
config :elixir, :dbg_callback, {MyMod, :debug_fun, [:stdio]}

Where you've defined a function MyMod.debug_fun/4.

Then when dbg/2 is called, your custom function will be called with three arguments prepended to any arguments you specify in your app config:

An AST representing the code to be debugged that was passed to dbg/2 as a first argument.
An AST representing any options given to dbg/2 as a second argument.
The Macro.Env struct representing the context in which dbg/2 was invoked.

You can find a more detailed example of this dbg/2 custom configuration approach in the docs.

Now that we've taken a pretty comprehensive look at what dbg/2 has to offer, let's move on to some other new Elixir 1.14 features that improve the development experience.

Inspect Improvements

Elixir 1.14 has improved the Inspect protocol implementation for a few core library modules: MapSet, Version.Requirement, and Date.Range. Previously, calls to inspect a new MapSet would return notation that is not valid Elixir:

iex> MapSet.new([1, :two, {"three"}])
#MapSet<[1, :two, {"three"}]>

This output can't be copied and pasted in IEx, or operated on by subsequent function calls.

This is no longer the case in Elixir 1.14. Now, you'll see this behavior:

iex> MapSet.new([1, :two, {"three"}])
MapSet.new([1, :two, {"three"}])

Here, the output is valid Elixir that can be operated on by subsequent Elixir code.

Some additional improvements have been made to Inspect protocol for structs:

When you inspect a struct, fields will be displayed in the order in which they are defined in defstruct.
If you want to customize the Inspect behavior for your struct by deriving it, you can use the new :optional keyword to omit struct fields that have not changed from their default value when displaying the inspection results. This simplifies struct representation and adds some much-needed brevity to the representation of larger structs. You can learn more info about the Inspect customization option in the docs.

We have just one more Elixir 1.14 improvement to look at before we wrap up.

Better Binary Evaluation Errors

In keeping with the improved debugging and developer experience we see throughout the Elixir 1.14 release, the latest version of Elixir also improves error messaging on binary construction and evaluation. This brings Elixir in line with Erlang/OTP 25 improvements in the same area.

Instead of getting generic ArgumentError messages when constructing binaries in an invalid manner, we get more informative error messages like this:

iex> 123 <> "456"
** (ArgumentError) expected binary argument in <> operator but got: 123

Want to learn how you can also save time debugging an app in production with AppSignal? Check out how Upside does it.

Wrap Up

Elixir has a devoted following and an increasingly growing community and adoption rate, in part because the language prioritizes developer happiness.

This latest Elixir release is no exception. With new debugging capabilities and improvements to Inspect protocols as well as binary evaluation error messages, Elixir's developer experience toolkit has grown even more.

Thanks to these new features, you'll be more productive in Elixir than ever before. Upgrade your Elixir apps today, and get going!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Benchmark Your Elixir App's Performance with Benchee

Roy Tomeij — 2022-09-06T00:00:00+00:00

At some point, every software engineer will find themselves in a situation where they need to benchmark system performance and test the limits of what a given system can handle. This is a common problem in software engineering, and even more so in the applications that are well suited for Elixir.

Finding bottlenecks early on in an application can save a lot of time, money, and effort in the long run, and give developers confidence in the upper limit of a system.

In this post, we will introduce a tool called Benchee to benchmark parts of an Elixir application. We will also show you how to integrate Benchee with your automated test suite.

By the end of the article, you'll understand Benchee's full functionality and capabilities, and will be able to use it to measure your application's performance.

Let's get going!

Why Do I Need to Benchmark My Elixir Application?

Benchmarking, in a nutshell, is the process of measuring the performance of a system under specific loads or conditions. As part of the benchmarking process, you will be able to identify potential bottlenecks in your system and areas to improve.

For example, we can use benchmarking tools to answer questions like:

Can a system handle ten times the load of normal traffic?
Can the system run on a smaller infrastructure to handle the same load?
How long does it take to process 10, 100, 1,000, or 10,000 requests? Does the processing time scale linearly with the number of requests?

Just answering these questions alone can help your team avoid costly mistakes and proactively identify areas for improvement, avoiding downtime and unhappy users.

Benchmarking doesn't have to be expensive or time-consuming; it can be simple to get the right tools in place and make them part of an application's natural development life cycle.

What Is Benchee?

This is where Benchee comes in. Benchee is a tool that you can use to benchmark parts of an Elixir application. It is versatile and extensible, with more than a few plugins to enhance its functionality.

Prerequisites

Elixir Environment

To follow along, you will need to locally install Elixir and Phoenix. The easiest way to do so is to follow the official Elixir instructions, which will give you a couple of options for:

Local installation on Linux, Windows, and macOS
Dockerized versions of Elixir
Package manager version setups

I recommend a local installation for the best results.

Setting Up Our Elixir Application

For this article's purposes, we will set up a simple Elixir application that can calculate the Fibonacci sequence.

Start by creating a new application with mix new fibonacci_benchmarking:

As output, you will see the following:

* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/fibonacci_benchmarking.ex
* creating test
* creating test/test_helper.exs
* creating test/fibonacci_benchmarking_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd fibonacci_benchmarking
    mix test

Run "mix help" for more commands.

Next, in your favorite editor, add the following code to the lib/fibonacci_benchmarking.ex file:

defmodule FibonacciBenchmarking do
  def list(number), do: Enum.map(0..number, &fibonacci/1)

  def fibonacci(0), do: 0
  def fibonacci(1), do: 1
  def fibonacci(n), do: fibonacci(0, 1, n-2)

  def fibonacci(_, prv, -1), do: prv
  def fibonacci(prvprv, prv, n) do
      next = prv + prvprv
      fibonacci(prv, next, n-1)
  end
end

Note: The original code can be found in rosettacode.org.

Go to the fibonacci_benchmarking directory and run the following commands:

mix deps.get
iex -S mix

And once inside the elixir shell, you can run this:

iex(1)> FibonacciBenchmarking.list(10)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

If you see the above output, you have successfully set up your application, and are ready to proceed with Benchee.

Implement Benchmarking on an Elixir Application

First, we will need to install Benchee. Start by adding the following to your mix.exs file:

defp deps do
  [
    {:benchee, "~> 1.0", only: :dev}
  ]
end

Next, run the following command:

mix deps.get

We can validate that Benchee is installed by running the Elixir shell and the following snippet:

Benchee.run(%{
    "10_seq" => fn -> FibonacciBenchmarking.list(10) end
})

Benchee might take a second or two to warm up, but on completion, you should see the following output:

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Number of Available Cores: 8
Available memory: 62.76 GB
Elixir 1.13.3
Erlang 24.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 7 s

Benchmarking 10_seq ...

Name             ips        average  deviation         median         99th %
10_seq      973.04 K        1.03 μs  ±2735.15%        0.77 μs        1.60 μs

The code above makes a call to FibonacciBenchmarking.list(10), and Benchee measures the time it takes to execute the function.

Let's take a moment to understand the output of Benchee. By default, Benchee will output the following information:

ips stands for iterations per second. This number represents how many times a given function can be executed in a second. Higher is better.
average is the average time it takes to execute the function. Lower is better.
deviation is the standard deviation of the results. This is a measure of how much the results deviate from the average.
median is the middle value of the results.
99th % - 99% of all the measured values are less than this value.

While running Benchee in this fashion can be useful for ad-hoc benchmarks, a much better method is to include Benchee as part of our unit tests.

Automate Benchee for Elixir and Run Tests

By default, all Elixir and Phoenix applications have a test directory and use ExUnit to run tests. Our goal is to get Benchee running as part of our test suite and test a different implementation of the Fibonacci sequence.

Start by creating a new file called test/benchee_unit_test.exs, and copy the following code into it:

defmodule BencheeUnitTest do
  use ExUnit.Case
  alias Application.TestHelper

  @tag :benchmark
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "case_10_numbers" => fn() ->
        FibonacciBenchmarking.list(10)
      end
    })

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

Go ahead and run mix test on the console. Validate that the output looks like the following:

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Number of Available Cores: 8
Available memory: 62.76 GB
Elixir 1.13.3
Erlang 24.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 7 s

Benchmarking case_10_numbers ...

Name                      ips        average  deviation         median         99th %
case_10_numbers        1.31 M      765.77 ns  ±3552.51%         599 ns        1165 ns
.

Finished in 10.7 seconds (0.00s async, 10.7s sync)
1 test, 0 failures

Randomized with seed 612867

So far, we have integrated Benchee into our test suite and added the first test to validate one of the test cases. Let's add the second test case to compare. Update the test function to:

defmodule BencheeUnitTest do
  use ExUnit.Case
  alias Application.TestHelper

  @tag :benchmark
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "case_10_numbers" => fn() ->
        FibonacciBenchmarking.list(10)
      end,
      "case_1000_numbers" => fn() ->
        FibonacciBenchmarking.list(1000)
      end
    })

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

Just like we did before, we can run the test suite with mix test, and validate that the output looks like the following:

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Number of Available Cores: 8
Available memory: 62.76 GB
Elixir 1.13.3
Erlang 24.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s

Benchmarking case_1000_numbers ...
Benchmarking case_10_numbers ...

Name                        ips        average  deviation         median         99th %
case_10_numbers          1.33 M     0.00075 ms  ±3419.36%     0.00059 ms     0.00109 ms
case_1000_numbers     0.00010 M       10.21 ms    ±11.54%        9.83 ms       15.29 ms

Comparison:
case_10_numbers          1.33 M
case_1000_numbers     0.00010 M - 13598.03x slower +10.21 ms
.

Our second test scenario tries to compare the performance of the Fibonacci sequence with a list of 1,000 numbers; however, this is not a very practical way to test with multiple inputs. We can take advantage of the Benchee.run hooks and provide a list of inputs for each scenario.

Go ahead and open the test/benchee_unit_test.exs file and replace the contents with this code:

defmodule BencheeUnitTest do
  use ExUnit.Case
  alias Application.TestHelper

  @tag :benchmark
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "generate_list" => fn(input) ->
        FibonacciBenchmarking.list(input)
      end
    },
    inputs: %{
      "case_10" => 10,
      "case_100" => 100,
      "case_1000" => 1000,
      "case_10000" => 10000,
      "case_100000" => 100000
    })

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

In this new version of the code, we have generalized our generate list case to accept a list of inputs, and we can now run the test suite with mix test.

However, because of the size of our last input, you will get a message like this:

❯ mix test

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Number of Available Cores: 8
Available memory: 62.76 GB
Elixir 1.13.3
Erlang 24.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: case_10, case_100, case_1000, case_10000, case_100000
Estimated total run time: 35 s

Benchmarking generate_list with input case_10 ...
Benchmarking generate_list with input case_100 ...
Benchmarking generate_list with input case_1000 ...
Benchmarking generate_list with input case_10000 ...
Benchmarking generate_list with input case_100000 ...


  1) test benchmark fibonacci list generation (BencheeUnitTest)
     test/benchee_unit_test.exs:6
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x" (accepts :infinity)
       2. per test module by setting "@moduletag timeout: x" (accepts :infinity)
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. by running "mix test --timeout x" which sets timeout
       5. or by running "mix test --trace" which sets timeout to infinity
          (useful when using IEx.pry/0)

     where "x" is the timeout given as integer in milliseconds (defaults to 60_000).

     code: output = Benchee.run(%{
     stacktrace:
       (elixir 1.13.3) lib/task.ex:794: Task.await/2
       (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
       (benchee 1.1.0) lib/benchee/benchmark/runner.ex:77: Benchee.Benchmark.Runner.parallel_benchmark/2
       (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
       (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
       (benchee 1.1.0) lib/benchee/benchmark.ex:103: Benchee.Benchmark.collect/3
       (benchee 1.1.0) lib/benchee.ex:48: Benchee.run/2
       test/benchee_unit_test.exs:8: (test)
       (ex_unit 1.13.3) lib/ex_unit/runner.ex:500: ExUnit.Runner.exec_test/1
       (stdlib 3.17) timer.erl:166: :timer.tc/1
       (ex_unit 1.13.3) lib/ex_unit/runner.ex:451: anonymous fn/4 in ExUnit.Runner.spawn_test_monitor/4



Finished in 60.0 seconds (0.00s async, 60.0s sync)
1 test, 1 failure

Randomized with seed 567196

As it happens, we hit a timeout error after 60 seconds. Fortunately, as part of the stack trace, we get a couple of suggestions on how to solve this problem. For now, update the test suite with this code:

defmodule BencheeUnitTest do
  use ExUnit.Case
  alias Application.TestHelper

  @tag :benchmark
  @tag timeout: :infinity
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "generate_list" => fn(input) ->
        FibonacciBenchmarking.list(input)
      end
    },
    inputs: %{
      "case_10" => 10,
      "case_100" => 100,
      "case_1000" => 1000,
      "case_10000" => 10000,
      "case_100000" => 100000
    })

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

Note: Depending on your system, running that last scenario will take a while; feel free to remove it to continue with the tutorial.

Now that we have a baseline of our Fibonacci sequence generator's performance, a common and useful exercise is to compare the performance of different implementations of the same algorithm. In this case, we have an alternative implementation of the Fibonacci sequence generator based on a recursive function.

Start by updating the lib/fibonacci_benchmarking.ex file with the following code:

defmodule FibonacciBenchmarking do
  def list(number), do: Enum.map(0..number, &fibonacci/1)
  def list_alternate(number), do: Stream.unfold({0,1}, fn {a,b} -> {a,{b,a+b}} end) |> Enum.take(number)

  def fibonacci(0), do: 0
  def fibonacci(1), do: 1
  def fibonacci(n), do: fibonacci(0, 1, n-2)

  def fibonacci(_, prv, -1), do: prv
  def fibonacci(prvprv, prv, n) do
      next = prv + prvprv
      fibonacci(prv, next, n-1)
  end
end

Following that, we will update the test/benchee_unit_test.exs file to account for both implementations:

defmodule BencheeUnitTest do
  use ExUnit.Case

  @tag :benchmark
  @tag timeout: :infinity
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "generate_list_enum" => fn(input) ->
        FibonacciBenchmarking.list(input)
      end,
      "generate_list_stream" => fn(input) ->
        FibonacciBenchmarking.list_alternate(input)
      end
    },
    inputs: %{
      "case_10" => 10,
      "case_100" => 100,
      "case_1000" => 1000,
      "case_10000" => 10000,
    })

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

The update test case will run two scenarios side by side for each of the prescribed inputs, letting us compare their overall performance. Go ahead and run mix test to see the results.

Compiling 1 file (.ex)
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Number of Available Cores: 8
Available memory: 62.76 GB
Elixir 1.13.3
Erlang 24.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: case_10, case_100, case_1000, case_10000
Estimated total run time: 56 s

Benchmarking generate_list_enum with input case_10 ...
Benchmarking generate_list_enum with input case_100 ...
Benchmarking generate_list_enum with input case_1000 ...
Benchmarking generate_list_enum with input case_10000 ...
Benchmarking generate_list_stream with input case_10 ...
Benchmarking generate_list_stream with input case_100 ...
Benchmarking generate_list_stream with input case_1000 ...
Benchmarking generate_list_stream with input case_10000 ...

##### With input case_10 #####
Name                           ips        average  deviation         median         99th %
generate_list_stream        1.66 M      601.29 ns  ±3858.34%         441 ns         955 ns
generate_list_enum          1.35 M      742.83 ns  ±3201.26%         610 ns        1164 ns

Comparison:
generate_list_stream        1.66 M
generate_list_enum          1.35 M - 1.24x slower +141.55 ns

##### With input case_100 #####
Name                           ips        average  deviation         median         99th %
generate_list_stream      258.49 K        3.87 μs   ±512.17%        3.29 μs        8.04 μs
generate_list_enum         31.71 K       31.54 μs    ±26.84%       30.57 μs       41.17 μs

Comparison:
generate_list_stream      258.49 K
generate_list_enum         31.71 K - 8.15x slower +27.67 μs

##### With input case_1000 #####
Name                           ips        average  deviation         median         99th %
generate_list_stream       17.67 K      0.0566 ms    ±14.14%      0.0550 ms      0.0988 ms
generate_list_enum         0.102 K        9.84 ms     ±9.92%        9.60 ms       14.14 ms

Comparison:
generate_list_stream       17.67 K
generate_list_enum         0.102 K - 173.90x slower +9.78 ms

##### With input case_10000 #####
Name                           ips        average  deviation         median         99th %
generate_list_stream        501.16      0.00200 s    ±33.05%      0.00190 s      0.00370 s
generate_list_enum            0.27         3.75 s    ±12.05%         3.75 s         4.06 s

Comparison:
generate_list_stream        501.16
generate_list_enum            0.27 - 1876.93x slower +3.74 s
.

Finished in 65.3 seconds (0.00s async, 65.3s sync)
1 test, 0 failures

Randomized with seed 839542

As you can see, our list function's Enum implementation is much slower than the Stream implementation, especially when the input size is larger. Comparing the performance of the two implementations is valuable in understanding the trade-offs and will help you develop more performant applications.

When adding benchmarking tests to parts of your automated testing, consider the potential drawbacks, such as the increased time it takes to run the tests. In this case, the benchmarking tests are tagged with :benchmark and can be excluded from the default test suite. This allows us to run the benchmarking tests separately from the unit tests and only when we need to.

A much better approach is to take advantage of CI/CD pipeline integration like GitHub Actions and run the benchmarking tests as part of the pull request validation process. This way, we can run the benchmarking tests as part of the CI/CD pipeline and get the results without having to run the tests locally.

Improving Benchee Reporting

Now, while seeing results on the console can be useful for a quick glance, the console is not the most convenient way to share results with your team. Benchee provides a number of different ways to export your results to a file.

For this example, we will use benchee_html to generate an HTML report with our benchmarking test results. To do this, we will add the benchee_html dependency to our mix.exs file:

def deps do
  [
    ...
    {:benchee_html, "~> 1.0", only: [:dev, :test]}
  ]
end

Next, we will update the test/benchee_unit_test.exs file to generate the HTML report:

defmodule BencheeUnitTest do
  use ExUnit.Case

  @tag :benchmark
  @tag timeout: :infinity
  test "benchmark fibonacci list generation" do
    # capture benchee output to run assertions
    output = Benchee.run(%{
      "generate_list_enum" => fn(input) ->
        FibonacciBenchmarking.list(input)
      end,
      "generate_list_stream" => fn(input) ->
        FibonacciBenchmarking.list_alternate(input)
      end
    },
    inputs: %{
      "case_10" => 10,
      "case_100" => 100,
      "case_1000" => 1000,
      "case_10000" => 10000,
    },
    formatters: [
      Benchee.Formatters.HTML,
      Benchee.Formatters.Console
    ])

    results = Enum.at(output.scenarios, 0)
    assert results.run_time_data.statistics.average <= 50_000_000
  end
end

Let's go ahead and run the tests again:

mix test

On completion, you should see the following report open in your browser:

The HTML report provides a much more detailed view of the benchmarking results and allows us to share results with our team easily. For example:

In addition to the HTML report, Benchee also supports exporting results to JSON, CSV, and XML formats. Exporting results to a file is a great way to integrate them with automation, such as CI/CD pipelines.

Monitoring Your Elixir App in Production

Benchee can help you discover potential performance bottlenecks, but what about how fast things really are in your production app?

To be able to discover new and existing bottlenecks, and solve bugs and other issues your users may face, you need to use an APM. AppSignal has been supporting Elixir developers for years and seamlessly integrates with your app. Bonus: We're the only APM that ships stroopwafels to new users 😎

Wrapping Up and Next Steps

In this tutorial, we discovered how to benchmark Elixir applications with the Benchee library.

We also learned how to compare the performance of different implementations of the same algorithm.

Yet we have only scratched the surface of Benchee's capabilities. As a next step, I highly encourage you to explore the available Benchee configuration options and visualization plugins.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Predictable Code in Elixir: Expressions as Reducers and Macros

Roy Tomeij — 2022-08-23T00:00:00+00:00

In the first part of this series on maintainable Elixir code, we started by applying rules for code predictability in our code design. We immediately saw an emerging pattern: a series of transformations on state. In this part, we'll explore this pattern further.

We'll first learn how to write expressions as reducers. Then we'll use metaprogramming to make use of reducers and enforce code style seamlessly.

Finishing up, we'll see an example where all the pieces fit together.

Let's get going!

Quick Recap of Part One and What We'll Do

In the first part, we experimented with applying code styles to this snippet:

with %Payment{} = payment <- fetch_payment_information(params),
     {:ok, user} <- Session.get(conn, :user),
     address when !is_nil(address) <- fetch_address(user, params),
     {:ok, order} <- create_order(user, payment, address) do
  conn
  |> put_flash(:info, "Order completed!")
  |> render("checkout.html")
else
  {:error, :payment_failed} ->
    handle_error(conn, "Payment Error")

  %Store.OrderError{message: message} ->
    handle_error(conn, "Order Error")

  error ->
    handle_error(conn, "Unprocessable order")
end

By the end of this post, we'll have all the tools to rewrite it like this:

case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end

While writing this post, I created ex_pipeline, an open source library that allows developers to create pipelines while enforcing code style easily.

All code examples in this post are simplified versions of what exists in that project.

Chain of Transformations

One of my favorite hobbies is woodworking — I'm not good at it, by all means, but it's something that I do enjoy. Before starting a new project, I always write down all the materials and tools I'll need, as well as the steps I'll follow. That's how I go from pieces of wood to a finished table, from an initial state to a finished state.

Let's think about how a table is made: you get some raw materials — wood, nails, glue — and a few tools. Then you execute a set of actions to modify the shape and appearance of the materials — cutting, assembling, polishing, and painting.

You apply a chain of transformations on an initial state (raw materials) until you get the desired state (the table).

We already talked about the emerging pattern of a pipeline. That's what we are going to explore over the next paragraphs.

Writing Expressions as Reducers in Elixir

Reducers are one of the key concepts of functional programming. They are very simple to understand: you take a list of values, apply a function to each element, and accumulate the results.

In Elixir, we can work with reducers using the Enum.reduce/3 function and its variants — Enum.reduce/2 and Enum.reduce_while/3.

For example, we can use a reducer to filter all even numbers from a list:

numbers = [1, 2, 3, 4, 5, 6]
starting_value = []

Enum.reduce(numbers, starting_value, fn current_number, filtered_values ->
  if rem(current_number, 2) == 0 do   # is the current number even?
    [number | filtered_values]        # if so, add it to the accumulator
  else                                # otherwise...
    filtered_values                   # just ignore it and return the accumulator
  end
end)

This code will return the list [2, 4, 6].

To execute different actions on a value, we can transform the list of values into a list of functions. Then we pass the value as the initial value for the accumulator:

transformations = [
  fn x -> x + 1 end, # add 1
  fn x -> x * 2 end  # multiply by two
]
starting_value = 10

Enum.reduce(transformations, starting_value, fn transformation, current_value ->
  apply(transformation, [current_value])
end)

This looks like a complicated way to write (10 + 1) * 2, right? But by expressing each transformation as a function, we can attach an arbitrary number of transformations. They all must accept the same number of parameters and return the next updated state. In other words, these functions must look the same and be small.

In the end, we want to write the previous example as something like this:

defmodule MyModule do
  use Pipeline

  def sum_step(value, _), do: {:ok, value + 1}
  def double_step(value, _), do: {:ok, value * 2}
end

In the next part, we can do just that using a bit of magic from macros!

Sewing Reducers Into Features with Macros

Now that we know how to use reducers let's use them to write pipelines.

The goal is to make the process of writing, reading, and maintaining pipelines as easy as possible. So, instead of adding a DSL or using complex configs, we just want to write our functions using the same pattern of name and arity. We can then automate the process of building the pipeline by detecting which functions follow this pattern.

We create a new pipeline by writing code on the same pattern.

The State: Track What's Happening in Elixir

Before we dive into macros, let's take a step back and talk a little bit about the state of the pipeline.

When executing a pipeline, we need to know a few things: the initial state, the current state, if it's still valid, and if there are any errors. All this information will help us debug and troubleshoot any problems we might find when writing our code.

Also, by abstracting state updates in this module, we can enforce a very important rule on reducers: the return value must be an ok/error tuple.

The starting point of a module that manages state should be something like this:

defmodule Pipeline.State do
  defstruct [:initial_value, :value, :valid?, :errors]

  def new(value) do
    %Pipeline.State{valid?: true, initial_value: value, value: value, errors: []}
  end

  def update(state, {module, function_name} = _function_definition, options \\ []) do
    case apply(module, function_name, [state.value, options]) do
      {:ok, value} ->
        %Pipeline.State{state | value: value}

      {:error, error_description} ->
        %Pipeline.State{state | valid?: false, errors: [error_description | state.errors]}

      bad_return ->
        raise "Unexpected return value for pipeline step: #{inspect(bad_return)}"
    end
  end
end

The new/1 function will create a new valid and clean %Pipeline.State{} struct based on the initial value we give to the pipeline executor.

The update/3 function will update or invalidate the given state by calling function with the state's value and the given options.

If the given function returns an {:ok, <updated_value>}, then the state's value is updated, and we are good to go. However, if the function returns an {:error, <error>} value, the state is invalidated. The returned error is added to the list of errors in the state. If the function returns anything that's different from an ok/error tuple, it will raise an exception.

See the full implementation of this module.

Pipeline Steps and Hooks

Each function that transforms the state will be called a step. Here are some guidelines to follow:

A step is a function whose name ends with _step and accepts two parameters.
Steps execute in the order they are declared in their module.
If one step fails, then all remaining steps are ignored.

Sometimes, we still want to execute code, even during failures — write a log, update a trace, publish a message, etc. We have a special type of step for these cases: a hook. Let's list the guidelines for hooks:

A hook is a function whose name ends with _hook and accepts two parameters.
Hooks execute in the order they are declared, just like steps.
They are always executed at the end of the pipeline, after all steps execute.

Good! But how do we detect that steps and hooks exist? The answer is metaprogramming! Luckily for us, Elixir has powerful metaprogramming tools to make all this possible.

Setting Up the Required Macros in Elixir

We'll need two things: macros and compile callbacks to read information about the functions of a module, and store what a step and a hook are.

Starting with the base module Pipeline, we want to use Pipeline in other modules. For that to happen, we declare a special macro, __using__/1. This macro is called with the keyword use in Elixir.

In this macro, we add a compile hook to the target module so that we can inject code into it. The @before_compile hook does exactly that by calling the given function when a compiler is about to generate the underlying bytecode. It accepts either a module or a tuple with a module and function name. We'll go with the second option to make things simpler.

defmodule Pipeline do
  defmacro __using__(_options) do
    quote do
      # Calls Pipeline.build_pipeline/1 before generating the final bytecode, allowing us to analyze, generate and
      # inject code whenever we call `use Pipeline`.
      @before_compile {Pipeline, :build_pipeline}
    end
  end

  defmacro build_pipeline(env) do
    # inject code!
  end
end

Anything called inside the quote block will be injected into the caller module — including the result of the @before_compile hook.

Check out the official docs for more information about quote and unquote blocks.

Now, whenever we want to create a pipeline, we call use:

defmodule MyPipeline do
  use Pipeline # <- code will be injected!
end

Detecting Steps and Hooks

We have everything in place to inject code, but we still need to generate the code that will be injected. Let's expand the build_pipeline/1 macro:

defmacro build_pipeline(env) do
  # Fetch all public functions from the caller module
  definitions = Module.definitions_in(env.module, :def)

  # Filter functions that are steps and hooks
  steps = filter_functions(env.module, definitions, "_step", 2)
  hooks = filter_functions(env.module, definitions, "_hook", 2)

  # Generate code on the caller module
  quote do
    # returns all steps and hooks
    def __pipeline__, do: {unquote(steps), unquote(hooks)}

    # Syntatic sugar so we can execute directly from the module that defines it
    def execute(value, options \\ []), do: Pipeline.execute(__MODULE__, value, options)
  end

  # Filter functions from the given module based on their suffix and arity
  defp filter_functions(module, definitions, suffix, expected_arity) do
    functions =
      Enum.reduce(definitions, [], fn {function, arity}, acc ->
        # Detect if the function name ends with the desired suffix: _step or _hook
        valid_name? =
          function
          |> Atom.to_string()
          |> String.ends_with?(suffix)

        # Detect if the function arity matches the expected arity
        has_expected_args? = arity == expected_arity

        cond do
          valid_name? and has_expected_args? ->
            # In this case, the function does end with the desired suffix and has the correct arity.
            # We include it in the accumulator along with the line where it was declared so we can
            # order them by their position in their module
            {_, _, [line: line], _} = Module.get_definition(module, {function, arity})
            [{module, function, line} | acc]

          valid_name? ->
            # If the function has the desired suffix but the arity is not correct, we raise this error
            # because we don't want people trying to figure out why their steps or hooks are not
            # being executed.
            raise(PipelineError, "Function #{function} does not accept #{expected_arity} parameters.")

          true ->
            # If the function doesn't match our filters, it's not a part of the pipeline and can be
            # just ignored
            acc
        end
      end)

    # At this point we have filtered the functions that match the filter criteria but they
    # are not in order. We use the line number information to sort them and then drop it
    # once we don't need it anymore.
    functions
    |> Enum.sort(fn {_, _, a}, {_, _, b} -> a <= b end) # order by line number
    |> Enum.map(fn {m, f, _l} -> {m, f} end)            # drop line number
  end
end

This macro receives the compiler env, which has lots of context information. Here, the important bit is the caller module, accessible via the env.module attribute.

Then we use the module.definitions_in/2 special function to get a list of all functions declared on the module env.module with the def keyword.

We call the Pipeline.filter_functions/4 function to filter these definitions by their suffix and arity, essentially detecting our steps and hooks! The Pipeline.filter_functions/4 function is kind of big, so I've added comments to help you navigate through it.

As said before, anything inside the quote block is injected directly into the caller module. So whenever we call use Pipeline, the module will have two extra functions: __pipeline__/0, and execute/2.

The Pipeline module uses the first function to execute our custom pipeline, while the execute/2 function is just a convenience function that will execute Pipeline.execute/3. It allows us to execute our pipeline by calling MyPipeline.execute(value, options).

Speaking of Pipeline.execute/3, it's time to define it. This function is the core of this engine, and it's responsible for actually calling a reducer that will power our pipeline engine:

defmodule Pipeline do
  # ...

  def execute(module, value, options) do
    # Build a new initial state of the pipeline
    initial_state = State.new(value)

    # Fetch steps and hooks from the module
    {steps, hooks} = apply(module, :__pipeline__, [])

    # Run each step from the pipeline and build the final state
    final_state =
      Enum.reduce(steps, initial_state, fn reducer, current_state ->
        State.update(current_state, reducer, options)
      end)

    # Run each hook with the final state
    Enum.each(hooks, fn {module, function} ->
      apply(module, function, [final_state, options])
    end)

    # Transform the state value into an ok/error tuple
    case final_state do
      %State{valid?: true, value: value} ->
        {:ok, value}

      %State{errors: errors} ->
        {:error, errors}
    end
  end
end

Check out the complete version of this module here.

Note: In the complete version of this module, there is also another kind of hook, called an async hook. Async hooks work just like the hooks presented here, but are executed in parallel rather than sequentially. They will not be covered in this post, since they are essentially a hook executed with a Task.

Building and Executing a Pipeline

By simply calling use Pipeline and writing function names with the _step or _hook suffix, we are now able to build our pipelines. Let's rewrite our last example from the first part of the post with this new mechanic:

defmodule Checkout do
  use Pipeline

  def fetch_payment_information_step(value, options) do
    # ...
  end

  def fetch_user_step(value, options) do
    # ...
  end

  def fetch_address_step(value, options) do
    # ...
  end

  def create_order_step(value, options) do
    # ...
  end

  def report_checkout_attempt_hook(state, options) do
    # ...
  end
end

We can execute this new pipeline like this:

case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end

No more with blocks, and no need to manually handle errors. The controller is now focused on handling the HTTP layer, and the checkout feature has an official place to live. Adding a new step to the checkout process just requires that we write a function in the right place!

Wrap Up

In this post, we learned how to express a set of transformations as a reducer. Then, using the power of macros, we automatically created pipelines by detecting the signature of functions at compile time.

The features that use pipeline mechanics are explicit and contained — we know exactly what to expect and where to go when modifying them. We expect all steps to look and behave in the same way. The basics of error handling are already in place, you just need to decide what to do. These pipelines are, indeed, predictable.

I hope you've found this two-part series about keeping your Elixir code maintainable helpful. Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Write a Standalone CLI Application in Elixir

Roy Tomeij — 2022-08-09T00:00:00+00:00

While Elixir is frequently associated with web development, this is not where its capabilities end. As a general-purpose language, it can be used for virtually anything. You don't have to take my word for it — projects such as Nerves, Nx, Scenic, or LiveBook speak for themselves.

But today, we will focus on something different: writing a command-line application in Elixir and preparing it for distribution.

Before we get going, let's touch on why we'd write a CLI app and what we'll cover in this tutorial.

Why Write a CLI App in Elixir?

CLI (command-line interface) applications have a wide variety of purposes. You probably already use some of them often — think of git, asdf, or even mix.

In our daily dev life, we usually write such applications to automate things that are otherwise tedious — formatting code, preparing deployments, running backups, synchronizing data, or... generating code. Yes, think about creating language-specific files from .proto definitions. You probably would use protoc or buf for that. Both of them are, in fact, CLI applications.

In this tutorial, we are going to build something a lot simpler. Let's set our requirements as follows: a script should take a path to a directory as an argument. We will (non-recursively) get all the files from there and calculate their mean size.

During this journey, you will also learn about tools that will help you achieve your goal. This kind of endeavor deserves a special name, so we will call it MFSC — Mean File Size Calculator.

ElixirScript: Let's Start Simple

Elixir already has a tool that is enough to get started. It's called ElixirScript and is really simple to use: you write code in a file with an .exs extension, and then you run it with elixir my_file.exs. I'm sure you have seen some .exs files before — for example, in Phoenix's config directory or in tests. These are examples of ElixirScript.

So, without further ado, let's create the first version of our MFSC:

defmodule MFSC do
  def call do
    [directory] = System.argv()

    with true <- File.dir?(directory),
         {:ok, files} <- list_files(directory) do
      Enum.reduce(files, 0, fn file, size -> size + calculate_file_size(file) end)
      |> Kernel.div(length(files))
      |> IO.puts()
    end
  end

  defp list_files(directory) do
    case File.ls(directory) do
      {:ok, files} ->
        files =
          files
          |> Enum.map(&Path.join(directory, &1))
          |> Enum.reject(&File.dir?/1)

        {:ok, files}

      error ->
        error
    end
  end

  defp calculate_file_size(file) do
    file
    |> File.stat!()
    |> Map.get(:size)
  end
end

MFSC.call()

We define a module called, unsurprisingly, MFSC with a call/0 function. This lists the files in the directory, then iterates on them to find their size before finally calculating the average. The only piece relevant to a command-line script is this line:

[directory] = System.argv()

System.argv/0 here returns a list of arguments passed to the script. We optimistically assume it's always just one argument and assign it to the directory variable.

That's it. If you run the script with an existing directory as an argument, it will return the mean length in bytes:

> elixir mfsc.exs ~/some_directory
1184
> elixir mfsc.exs ~/none_existing_dir
>

As you can see, for an incorrect input (a directory that does not exist), it will just return nothing. Let's fix it by adding else to the with statement.

with true <- File.dir?(directory),
     {:ok, files} <- list_files(directory) do
  Enum.reduce(files, 0, fn file, size -> size + calculate_file_size(file) end)
  |> Kernel.div(length(files))
  |> IO.puts()
else
  _ ->
    IO.puts("Invalid input")
    System.halt(1)
end

Note that I didn't only add an error message but also a System.halt(1) call. This is to return a non-zero (i.e., non-success) exit code from a script (you can check the exit code of the last run command with echo $?). It is important to do so to be a 'good CLI land citizen.'

Why? Command-line applications often do not work on their own but are chained together or called by other programs. An exit code helps to detect if a given part of a chain was run successfully (and we can expect the output to be in some predefined format) or not (in which case, we probably shouldn't even try to parse output).

We have a working script, except for one small issue. When you run it on an empty directory, it will crash because of division by zero:

** (ArithmeticError) bad argument in arithmetic expression: div(0, 0)
    :erlang.div(0, 0)
    mfsc.exs:8: MFSC.call/0
    (elixir 1.13.3) lib/code.ex:1183: Code.require_file/2

That's not nice. But what should we do on an empty directory? It's not obvious. We could return zero. Or alternatively, halt the application with a non-zero exit code.

Or maybe — just a thought — we should leave the choice to the user?

OptionParser: More Robust Input Handling in Elixir

Remember how we assumed that there would always be only one input argument to our script? We're about to ditch this assumption. We want to introduce an option for the user to decide what should happen when an empty directory is found.

By default, we want to return an error, but there will be a flag to alter this behavior:

> elixir mfsc.exs /my/empty/dir
Empty directory
> echo $?
10
> elixir mfsc.exs /my/empty/dir --allow-empty
0
> echo $?
0

Fortunately, Elixir has a tool readily waiting for us in its standard library: OptionParser.

Let's first play with it a bit in iex:

iex(1)> OptionParser.parse(["/my/empty/dir", "--allow-empty"], strict: [allow_empty: :boolean])
{[allow_empty: true], ["/my/empty/dir"], []}
iex(2)> OptionParser.parse(["/my/empty/dir"], strict: [allow_empty: :boolean])
{[], ["/my/empty/dir"], []}

The first argument to the OptionParser.parse/2 function is a list of arguments that we normally get from System.argv/0. In this case, we are simulating it manually.

The second one is the definition of our expected input. It is a keyword list with three possible keys: strict, aliases, and switches. For now, we will concentrate on strict as it is the basic one. In this example, we defined an expectation of one argument of the boolean type.

The return value is a tuple of three elements: the first is a list of parsed options, the second is for other (positional) arguments, and the third one is for errors. Let's see some more examples:

iex(3)> OptionParser.parse(["/my/empty/dir", "/tmp/second/dir"], strict: [allow_empty: :boolean])
{[], ["/my/empty/dir", "/tmp/second/dir"], []}
iex(4)> OptionParser.parse(["/my/empty/dir", "--allow-empty=3"], strict: [allow_empty: :boolean])
{[], ["/my/empty/dir"], [{"--allow-empty", "3"}]}

In the first example, we have two positional arguments and no parsed options. In the second, we violate the constraint that --allow-empty should be a boolean switch by passing the number 3 to it. As a result, it doesn't appear in parsed options but instead lands in errors.

Knowing how OptionParser works, we can add it to our app:

defmodule MFSC do
  def call do
    parsed =
      System.argv()
      |> OptionParser.parse(strict: [allow_empty: :boolean])

    with {options, [directory], []} <- parsed,
         true <- File.dir?(directory),
         {:ok, files} <- list_files(directory) do
      Enum.reduce(files, 0, fn file, size -> size + calculate_file_size(file) end)
      |> print_size(files, options)
    else
      _ ->
        IO.puts("Invalid input")
        System.halt(1)
    end
  end

  defp print_size(0, _, options) do
    case Keyword.get(options, :allow_empty) do
      true ->
        IO.puts("0")

      _ ->
        IO.puts("Empty directory")
        System.halt(1)
    end
  end

  defp print_size(total_size, files, _) do
    IO.puts(div(total_size, length(files)))
  end

  # rest of the code does not change
end

Nice! We support our first input option.

There is still a lot of work to do to make this app's interface friendlier, but I'll leave it up to you at this point. Some ideas on what you could do include:

Returning different error messages and exit codes for various errors (non-existing directory vs. empty directory without --allow-empty vs. some random access error)
An option to print a result in a different format than just the number of bytes, for example:

--output-format [B|K|M|smart]

Allowing recursive counting, i.e., also calculate file sizes in subdirectories and their subdirectories. Beware of symbolic links leading to cycles!

The Limitations of ElixirScript

So far, we've been using ElixirScript to create our CLI apps, but as our code grows, you will soon notice some limitations in this approach.

Splitting code into multiple files is inconvenient, and you cannot really use all the awesome packages published at hex. You might overcome this by using Mix.install/2 inline:

Mix.install([
  {:ecto_sql, "~> 3.8"}
])

# some code using Ecto

There is, however, one last problem: to run your application, people need to have Elixir installed on their computers. This is what makes ElixirScript a nice tool to run something along an existing larger Elixir application (like tests in a Phoenix project), but not for our use case.

Elixir is still not a standard language. Some system repositories might have really old versions of the language. Generally, the fewer dependencies the user needs to install themselves, the more likely they will actually try out your application.

How can we do better, then?

Using Escript for Your Elixir App

Escript is a built-in way to distribute whole Elixir applications as a single-file executable. This is, of course, one level better than using ElixirScript. Let's look closely at how to include it in our application.

To use escript, we need to have a regular mix project, not just a single file. But this is fine, as we want to split our application into multiple files and use dependencies anyway. Let's create a project then:

mix new mfsc

This will create a bunch of files in an mfsc directory. Among them is lib/mfsc.ex, where we will put the code we built in the previous section.

Theoretically, you should be able to run it via mix run now, but here's an unpleasant surprise: mix run resets System.argv/0, and you'll always get an empty array as the input. We are not going to focus on hacking this. Instead, we will introduce escript right away.

The first step: remove MFSC.call() from the bottom of the file — we only want to define a module for now.

Now we will add support for escript to our mix.exs file. In the project function that returns a keyword list of the project configuration, add a relevant line for escript:

def project do
  [
    app: :mfsc,
    # bunch of other things
    escript: [main_module: MFSC]
  ]
end

We are almost there. Now we need to refactor the call/0 function to the main/1 function accepting one argument — args — which will be the equivalent of System.argv/0.

defmodule MFSC do
  def main(args) do
    parsed =
      args
      |> OptionParser.parse(strict: [allow_empty: :boolean])

# rest of the file remains unchanged

That's all, folks! We build the escript file with mix escript.build. This will create an mfsc executable file in the project's root directory. We can call it like a regular executable:

> mix escript.build
Generated escript mfsc with MIX_ENV=dev
> ./mfsc ~/empty_dir
Empty directory
> ./mfsc ~/empty_dir --allow-empty
0

This is a huge step forward from ElixirScript — and the user does not need to have Elixir installed on their machine. Note, however, that they still need to have Erlang. On the other hand, this is a much more common dependency than Elixir, and chances are they already have it. In the worst-case scenario, it's just one thing less to install.

If that's not enough, we can improve things even further.

Bakeware and Burrito for Elixir

Bakeware is a relatively new project (created in mid-2020) aiming to create dependency-less executables from Elixir code. Let's try it with MFSC. Bakeware leverages Elixir releases, so we need to add support for them to mix.exs. On top of that, we need the bakeware dependency and to add our entry module to application. The complete mix.exs should look like this:

defmodule Mfsc.MixProject do
  use Mix.Project

  def project do
    [
      app: :mfsc,
      version: "0.1.0",
      elixir: "~> 1.13",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      escript: [main_module: MFSC],

      releases: [                          # here we define cli release
        cli: [
          steps: [:assemble, &Bakeware.assemble/1]
        ]
      ]
    ]
  end

  def application do
    [
      extra_applications: [:logger],
      mod: {MFSC, []}                      # adding our entry module
    ]
  end

  defp deps do
    [
      {:bakeware, "~> 0.2.4"}              # bakeware dependency
    ]
  end
end

After that, we need to adjust our MFSC module by adding Bakeware scripting support:

defmodule MFSC do
  use Bakeware.Script

  @impl Bakeware.Script
  def main(args) do

# rest of the file

And with that in place, we just need to run mix release cli. This will take some time, but we end up with our standalone CLI application with a _build/dev/rel/bakeware/cli file. Let's take it for a ride!

> _build/dev/rel/bakeware/cli ~/empty_dir
Empty directory
> _build/dev/rel/bakeware/cli ~/empty_dir --allow-empty
0

This definitely looks familiar. However, you should know one thing — these files will only work on a similar operating system as the one they were built on. For example, I created mine on Linux, and when I sent it to another Linux machine, it worked without any issues. But it did not work on Intel Mac, not to mention M1 Mac.

We might try to address this issue with an even newer project than Bakeware. It's called Burrito and the first public commit is just from October 2021. It promises cross-platform standalone releases and is aimed at CLI applications. Burrito has a few additional dependencies, namely Zig and xz. When we install them, we add another release to mix.exs (remember to add the burrito dependency too!):

burrito_cli: [
  steps: [:assemble, &Burrito.wrap/1],
  burrito: [
    targets: [
      macos: [os: :darwin, cpu: :x86_64],
      macos_m1: [os: :darwin, cpu: :aarch64],
      linux: [os: :linux, cpu: :x86_64],
      windows: [os: :windows, cpu: :x86_64]
    ],
  ]
]

Now, run the release process:

mix release burrito_cli

After a while, you should have the executables for different operating systems waiting under the burrito_out directory, for example, burrito_out/burrito_cli_linux. Note that Burrito is still a work in progress, and you may experience some bumps on the road. For me, at the time of writing this article, building for macOS from Linux did not work.

If you want to learn more about Burrito, there's a great talk by its author, Digit, from the EPEX conference. It's called Wrap your app in a BEAM burrito! In this talk, you'll learn why the project was born and the main hurdles along the way, followed by a live coding session.

The Downsides of Using Elixir for CLI Applications

Now that we've seen how easy it is to build a standalone CLI application in Elixir, let me warn you about one thing: the startup time will be slow. In my tests, it always takes about half a second.

This is not a problem for long-running or interactive applications but might be a showstopper if you need a small tool to execute multiple times — for example, the find command:

find . -maxdepth 1 -type f -exec ./my_elixir_app {} \;

The startup time of half a second is multiplied by the number of files, which may add up to quite significant delays. You can experience this pain in the real world when you generate protobuf files for Elixir with protobuf-elixir.

Should I Use Elixir for CLI Apps?

If some of the problems outlined in the previous sections of this post do not scare you or do not apply to your use case, by all means use Elixir for CLI applications. It can be a lot of fun, especially combined with its concurrency abilities.

I personally did in my simple stress-tester application, which spawns multiple processes and attempts to flood a given URL with requests, gathering statistics on how it goes. And I used it for work more than once, where a full-blown solution was not required.

Wrapping Up

In this post, we built a simple CLI application using ElixirScript, learned to use OptionParser, then released it using escript as an executable, which only requires Erlang to run.

After that, we looked into creating more standalone releases with Bakeware or Burrito. Finally, we explored some downsides of choosing Elixir for certain types of CLI tools.

I hope you found this a good starting point for writing a CLI app in Elixir.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Write a Functor in Elixir

Roy Tomeij — 2022-07-26T00:00:00+00:00

There’s a function called Enum.map in Elixir that works on multiple collection types, but it's not without its issues.

Note: The article is inspired by the Witchcraft library, which we covered in one of our previous posts.

But first: what's the problem with Enum.map exactly?

The Issue with `Enum.map` in Elixir

While Enum.map works fine with lists, its behavior is a bit odd for other collections.

iex(1)> cities = %{"Latvia" => "Riga", "France" => "Paris", "Germany" => Berlin}
%{"France" => "Paris", "Germany" => Berlin, "Latvia" => "Riga"}
iex(2)> cities = Enum.map(cities, fn x -> x end)
[{"France", "Paris"}, {"Germany", Berlin}, {"Latvia", "Riga"}]

Even though we call it with an identity function that returns its input unchanged, we don’t get the same structure back.

And we can’t use the result as a map.

iex(2)> Map.fetch(cities, "Latvia")
** (BadMapError) expected a map, got: [{"France", "Paris"}, {"Germany", "Berlin"}, {"Latvia", "Riga"}]
    (stdlib 3.17) :maps.find("Latvia", [{"France", "Paris"}, {"Germany", "Berlin"}, {"Latvia", "Riga"}])

Understanding how Enum.map works on maps depends on knowing two things:

Elixir thinks of maps as lists of tuples (but only when it wants to).
Enum.map will always return a list as a result.

Safe to say, this is not intuitive. But what could be a better way to do things, and how can we ensure we don’t make the same mistake again?

Let's see how a functor can help.

What’s a Functor in Haskell?

In Haskell, Functor is a typeclass, an ‘interface’ with a set of methods common to multiple data types.

-- Definition for illustrative purposes.

class Functor f where
  fmap :: (a -> b) -> f a -> f b
  (<$) :: a -> f b -> f a

The closest thing to typeclasses in Elixir is protocols. In the same way that we have Enumerable (Enum) in Elixir, you can also think of Functor as Functor-able, or, in more human language, Mappable.

The important method of the Functor typeclass in Haskell is fmap.

`fmap`

fmap takes a function and a structure, then returns the same structure with the function applied to the structure's content.

In other words, it implements a structure-preserving transformation.

fmap is similar to Enum.map, but there are some key differences:

fmap returns the structure it is called on, while Enum.map always returns a list.
It enables you to map a wider set of structures. For example, in Haskell, you can use fmap with single-item structures (like {:ok, result}) and even functions.

Note: In Haskell, fmap takes the function as the first argument. But since Elixir usually has data in the first position due to pipes, I've switched the arguments while adapting the concept.

It’s best to look at some examples to understand what fmap does. The examples below use the implementation that we will later create ourselves.

iex(1)> import Functor
Functor

Lists

iex(2)> fmap([1, 2, 3], fn x -> x + 1 end)
[2, 3, 4]

Maps

iex(3)> cities = %{"Latvia" => "riga", "France" => "paris", "Germany" => "berlin"}
%{"France" => "paris", "Germany" => "berlin", "Latvia" => "riga"}
iex(4)> fmap(cities, fn x -> String.capitalize(x) end)
%{"France" => "Paris", "Germany" => "Berlin", "Latvia" => "Riga"}

Trees

iex(5)> a = %Tree{value: 1, children: [%Tree{value: 2, children: []}, %Tree{value: 3, children: []}]}
%Tree{
  children: [%Tree{children: [], value: 2}, %Tree{children: [], value: 3}],
  value: 1
}
iex(6)> fmap(a, fn x -> x + 1 end)
%Tree{
  children: [%Tree{children: [], value: 3}, %Tree{children: [], value: 4}],
  value: 2
}

Result tuples

iex(7)> fmap({:ok, 4}, fn x -> x + 3 end)
{:ok, 7}
iex(8)> fmap({:error, :not_a_number}, fn x -> x + 3 end)
{:error, :not_a_number}

If you look at these examples, you’ll see that they all have a/some value(s) wrapped in a structure. fmap uses the function we provide to change the value(s) while keeping the structure the same.

If I could write its typespec, it would look something like this.

@spec fmap(f(a), (a -> b)) :: f(b)

It takes:

something of type a wrapped in type f
a function from type a to type b

And it returns a value of type b wrapped in type f.

Functor Laws in Elixir

fmap needs to follow a set of equational laws to achieve its goal. And if laws scare you — please skip this, look at the implementation, then come back, and it will be easier.

In Haskell, the compiler doesn't check these laws by default, but you need to follow them for the implementation to ‘make sense’.

The first law says that if you have a function that returns its output untouched, ‘fmapping’ it will do the same.

fmap(y, fn x -> x end) == y

The second law says that composing two ‘fmaps’ is the same as ‘fmapping’ the composition of those functions.

It looks something like this:

(fmap(x, f1) |> fmap (f2)) == (fmap(x, fn y -> f2(f1(y)) end))

The laws are awkward to express in Elixir, but they basically work to preserve the structure of the type you’re mapping over.

The implementation of Enum.map for maps, for example, satisfies the second law, but doesn’t satisfy the first. Hence, it is not a lawful implementation of fmap.

Why Do You Need to Know About Functors?

In programming, some patterns repeat themselves time and time again.

For a long time, people have been trying to describe these patterns and pass them on to other software developers to:

Solve problems faster
Expand ‘tools for thought’
Make communication easier.

A functor is also a pattern, just like singleton and factory.

Most classic patterns are useful, yet appear infrequently. But patterns like functors (and other math-based typeclasses) are so simple in their internal structure that they are bound to appear in your code (even if you don't mean them to). At that point, you can either use the knowledge we have to handle them or not.

Let's look at an example. In Elixir, a lot of our code is expressed in pipelines, which can be looked at as a series of transformations on data. It's very readable: one of the main reasons why the syntax of Elixir is so attractive.

Functors can help us make our code even better in two ways.

First, functors express pipe-able transformations of collections of data. If we have a fmap function that returns the same kind of wrapper, we can easily pipe transforms into each other without calling Enum.into() in between.

Secondly, functors also help to transform nested values in pipelines. If a function returns a data type with a more complex structure like {:ok, result}/{:error, error}, we sometimes run into problems when piping. To handle the value, we either need to deconstruct it or write a function that handles the structure and does the transformation behind the scenes.

The first can create such horrors as piping into case statements in the middle of a pipeline, while the second can introduce a lot of code duplication and make code more obscure.

fmap enables us to apply a function to a nested value without explicitly destructuring, unwrapping, and/or repacking the data. The word fmap informs other readers of your code what the function does. Then, after all transformations are complete, you can handle the resulting value.

Now that we have a feel for what functors are, we can try to replicate their behavior in Elixir. To do that, we’ll use protocols.

Simulating Functors in Elixir

In this section, we will create a protocol for functors and provide implementations for four data types: lists, maps, trees, and result tuples.

Set-up

For this tutorial, we will need a new Elixir project.

mix new functor

And a file called functor.ex.

Create a `Functor` Protocol

This is rather straightforward.

First we open functor.ex. Then we use the defprotocol macro to create a protocol.

We provide the functions that the protocol will have to the macro. In our case, the only function is fmap.

defprotocol Functor do
  def fmap(a, f)
end

That’s all we need to do.

Create an Implementation for Lists

Of course, a protocol is only as good as its implementations.

We can create new implementations for a protocol with the defimpl macro. Let’s do the list right inside functor.ex since list is a staple data type.

fmap for lists is the same as a regular map. So we’ll just use :lists.map from grandpa Erlang for the implementation.

defimpl Functor, for: List do
  @spec fmap(list, (list -> list)) :: list
  def fmap(a, f), do: :lists.map(f, a)
end

Here’s how fmap works for lists:

iex(1)> Functor.fmap([1,2,3], fn x -> x + 1 end)
[2, 3, 4]

Create an Implementation for Maps

Now we’ve come to the data type we mentioned at the start of the article.

The easiest way to make Enum.map() return a map is to pack it back into a map after mapping it.

defimpl Functor, for: Map do
  def fmap(a, f), do: Enum.map(a, f) |> Enum.into(%{})
end

But there’s a small problem here. It can easily fail.

iex(1)> Functor.fmap(%{Netherlands: "Amsterdam"}, fn {k, v} -> v end)
** (ArgumentError) argument error
    (stdlib 3.17) :maps.from_list(["Amsterdam"])
    (elixir 1.13.0) lib/enum.ex:1448: Enum.into_map/1

This is one of the reasons why Enum.map() on maps returns lists. The function you provide might create something that isn't a map anymore.

It’s also not a valid functor implementation. Functors can change only one value: they can map either keys or values, and we have to choose one for the implementation.

We can solve both of these problems at the same time by limiting the mapping operation to the values.

defimpl Functor, for: Map do
  def fmap(a, f) do
    map_in_list = Map.to_list(a)

    :lists.map(fn {k, v} -> {k, f.(v)} end, map_in_list)
    |> Enum.into(%{})
  end
end

Here’s how it works:

iex(1)> Functor.fmap(%{Netherlands: "Amsterdam"}, fn x -> x <> "!" end)
%{Netherlands: "Amsterdam!"}

There are some arguments against this implementation (even though it is a valid functor 😅). Having a map function work only on values might be as unintuitive as having it return a list.

You also can't work with the keys in any capacity — you need other functions for that. But every time you call fmap on a map, you will get a map back.

Create an Implementation for Trees

Now that we have seen how to define fmap for simpler data types, let's try to define it for a type not served by the Enum protocol — trees.

First off, we need to define the struct. We’ll do that in structures.ex.

defmodule Tree do
  defstruct value: nil, children: []

  @type t() :: %__MODULE__{
          value: any(),
          children: [t()]
        }
end

For simplicity’s sake, each of the tree's nodes has a value and a list of children that can be empty.

Now we can define the Functor implementation inside the module. In the definition, we can skip the for: Tree part — Elixir will know we mean the module struct.

defmodule Tree do
  # ...
  defimpl Functor do

  end
end

All that is left is to write the implementation, which will be recursive.

We’ll apply the function to the value if we have a leaf (which has no children).

def fmap(%Tree{value: value, children: []}, f), do: %Tree{value: f.(value), children: []}

If the node has children, we need to map this fmap over all of them. This gives us a great opportunity to use the Functor.fmap we created earlier!

def fmap(%Tree{value: value, children: children}, f) do
  updated_children = Functor.fmap(children, fn x -> fmap(x, f) end)
  %Tree{value: f.(value), children: updated_children}

If you understand the piece of code above, you have already achieved a certain level of enlightenment about functors ☯

Here’s how the function works:

iex(1)> a = %Tree{value: 1, children: [%Tree{value: 2, children: []}, %Tree{value: 3, children: []}]}
%Tree{
  children: [%Tree{children: [], value: 2}, %Tree{children: [], value: 3}],
  value: 1
}
iex(2)> Functor.fmap(a, fn x -> x + 1 end)
%Tree{
  children: [%Tree{children: [], value: 3}, %Tree{children: [], value: 4}],
  value: 2
}

As an exercise, you can try to create a struct for a binary tree — one where a node has a maximum of two children — and a fmap implementation for this struct.

Create an Implementation for Result Tuples

Finally, let's see how we can implement fmap for result tuples.

Now, mapping a tuple might seem counterintuitive for some. But transforming a collection of values is not the only thing fmap can do. Another angle of how we can look at fmap is that it lifts a function into a context.

In Elixir, functions sometimes return one of {:ok, result} or {:error, error}. We can call this the context of a computation that might have succeeded or failed.

Now imagine we want to apply a function such as fn x -> x + 1 end to this result.

We can either:

Unpack it via pattern-matching and handle the error right here, right now.
Lift the function inside the context of possible error and then send the result somewhere further.

It is pretty straightforward to do the second with this tuple. If we have an {:ok, result}, we just apply the function to the result. If we have an {:error, error}, we don’t apply the function, but simply pass on the error.

But there are some problems with making a functor implementation.

To do it, we have to dispatch on the Tuple type, which theoretically has a lawful Functor implementation of its own that changes only the last element of the tuple.

Since Elixir cannot discern our intentions, this would mean doing something practical yet a bit unlawful.

But since “practical yet a bit unlawful” sounds like the best characterization of Elixir I’ve read, we’ll go ahead and do it anyway.

defimpl Functor, for: Tuple do
  def fmap({:ok, result}, f), do: {:ok, f.(result)}
  def fmap({:error, reason}, \_f), do: {:error, reason}
end

Here’s how it works:

iex(2)> fmap({:ok, 4}, fn x -> x + 3 end)
{:ok, 7}
iex(3)> fmap({:error, :not_a_number}, fn x -> x + 3 end)
{:error, :not_a_number}

The lawful but unidiomatic way to do this in Elixir would be to create two structs — %Result.Ok{} and %Result.Err{} — and define the fmap implementation for those instead. As an exercise, you can try to do this on your own.

Are Functors Useful in Elixir?

Now that we have created a basic yet working protocol for functors, it’s good to ask: is this thing useful?

Well, it's kind of cute. We can use it to do some cool things, such as create a map function that works on both a list of trees and a tree of lists.

def mega_map(a, f) do
  Functor.fmap(a, fn x -> Functor.fmap(x, f) end)
end

And when we add a couple more implementations, having a mapping function that works on multiple data types (like Enum.map) but doesn't always return list can be handy.

But the power of Haskell-inspired ideas is proportional to the infrastructure you have to work with them. Let's think of this as an optimization exercise. We’re traveling away from a local idiomatic Elixir peak to write code that is more expressive and intuitive.

In Haskell, you have access to a ton of other ‘mathy’ typeclasses like Applicative, Monad, and more to help you write code that’s starkly different from Elixir. And it’s much easier because of the type system and other peripherals.

At the same time, what happens if we carefully pick out concepts from other programming languages and adapt them to how we write Elixir right now? We develop a way of writing Elixir that is more convenient and makes sense to adopters from other functional programming languages.

This is akin to mainstream languages borrowing higher-order functions and result types from functional programming. Haskell's whole complex type machinery is not needed in Elixir, but perhaps there is a simpler way to structure work with collections that doesn’t take inspiration from Ruby.

In particular, I've lately fallen in love with Witchcraft's vision for how the syntax of Elixir could look. It provides custom operators that work in a pipe-like fashion. For example, you can use ~> instead of |> fmap.

Having a few custom operators like these that synergize with how idiomatic Elixir is written, yet follow strict laws for implementation, would enable writing very expressive code that still feels like Elixir.

And given the news that Elixir is working on a set-theoretic type system, it’s high time to take a quick peek in the direction of other typed functional languages.

Wrap Up and Further Learning

In this article, we covered how to build a simple protocol for functors, together with implementations for four data types: lists, maps, trees, and result tuples.

If you’re interested in trying out more functors magic in Elixir, check out Witchcraft, which has been my main inspiration for this post.

And if you want to delve deeper into the world of typed functional programming, you should definitely try out Haskell. The best way to do that is to start with one of these books: Haskell Programming From First Principles or Get Programming With Haskell.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Writing Predictable Elixir Code with Reducers

Roy Tomeij — 2022-07-19T00:00:00+00:00

This is the first part of a two-part series about maintainable code in Elixir. In this part, we will show how code predictability plays a crucial role in a project's short and long-term health. We will use Elixir's built-in features for this, like the pipe operator, tuples, and with blocks.

First, we'll explain what predictability is and why it is so important. Then we will go through some tools that Elixir already has and how you can use them to write better code.

Finally, we'll demonstrate how some simple rules can help developers write code that is easy to read, write, and maintain.

Let's get started!

On Complexity and Predictability

I still remember when I first started learning about functions in my math class. Things seemed simple and easy to understand:

$$ L = \frac{1}{2} \rho v^2 S C_L $$

$$ y(x) = 2x $$

One function, one variable, and one simple operation.

But as the years passed and more complex things were added to the basics, the small and easy functions were gone. I was reading stuff like this:

Formula

f ( x , y ) = θ ˘ α lo g ( y 3 x 2 )

...What? Alpha? Theta with an inverted hat?

Eventually, though, after rereading and rewriting these kinds of functions over and over, they also become easy to work with.

In the excellent book The Programmer's Brain: What Every Programmer Needs to Know About, Felienne Hermans does an amazing job of explaining the different mechanisms that our brain uses to read, write, and understand code, as well as techniques to perform these tasks as efficiently as possible.

In short: the amount of new information that our brains can handle is actually pretty small, and there is not really much we can do about it.

The good news is that our memory is very different from a computer — and we can take advantage of that. With enough repetition, we eventually start to store patterns in our long-term memory and then use that knowledge to understand new information faster.

That's why a skill becomes easier with more practice, be it math, playing the guitar, or programming. The more we make things look and behave the same, the easier and faster it is to absorb them.

Why Should I Make My Elixir Code Predictable?

When it comes to understanding what we read and how well other people will understand the code that we write, being predictable is essential.

Three key elements make code predictable:

Structure: different blocks of code look and behave the same. That is the main reason it is easy to understand a for loop, even in languages that you've never used before.
Size: smaller blocks (like functions, modules, classes, etc.) are easier to remember.
Simplicity: less moving parts, like function parameters, help our brain keep track of what is happening.

One good example of predictability in the Elixir world is the Plug library because:

It uses behaviours to keep all implementations with the same structure (expected input and output).
Each plug is expected to be as small as possible since they add latency to the HTTP response.
They have a pretty simple shape: two functions, each with two parameters.

So, what if we make our entire code predictable?

Elixir's Tools for Writing Predictable Code

Elixir itself has a couple of nice tools that, when used correctly, help us create predictable code: its pipe operator and with statement.

Keep in mind that neither of these will enforce a pattern or design principle — they are just tools.

Elixir's Pipe Operator

The pipe |> operator is an amazing tool that helps us express a chain of function calls as a simple sequence of actions.

Even if you've never written any Elixir code, you probably understand what this piece of code is trying to do:

form_params
|> validate_form()
|> insert_user()
|> report()
|> write_response()

It's simple to read and easy to understand what is happening.

However, because it's so simple, we're also very limited in what we can do with it. Since all functions are chained, they depend on the previous result. If any of the functions break, there's not much we can do about it unless we add error handling to all of them.

For a proper pipeline where we can handle errors and don't want to make the functions dependent on one another, it's better to use the with statement.

Elixir's `with` Statement

Pipe operators are simple, but we often need to check the return to ensure we have a valid state.

Let's take the previous example and rewrite it with a with block:

with %User{} = user_data <- validate_form(form_params),
     {:ok, user} <- insert_user(user_data),
     :ok <- report(user) do
  write_response(user)
else
  error ->
    report(error)
    handle_error(error)
end

Now we have proper error handling and a chance to do something if our user data is invalid or it's not possible to insert the user. However, even in this simple example, it's a bit harder to read because of the added complexity. Note that since functions can return any value, we have to handle the return of each step explicitly.

Let's use a more complex example — a generic checkout code where we need information about the user, payment, and address — and try to create an order:

with %Payment{} = payment <- fetch_payment_information(params),
     {:ok, user} <- Session.get(conn, :user),
     address when !is_nil(address) <- fetch_address(user, params),
     {:ok, order} <- create_order(user, payment, address) do
  conn
  |> put_flash(:info, "Order completed!")
  |> render("checkout.html")
else
  {:error, :payment_failed} ->
    handle_error(conn, "Payment Error")

  %Store.OrderError{message: message} ->
    handle_error(conn, "Order Error")

  error ->
    handle_error(conn, "Unprocessable order")
end

Not only does each function have a different return type, but errors may also have different shapes.

Also, because the existing functions don't follow a pattern, a developer adding a new step to this feature will not know what the new function should return. A struct? ok/error tuple? A non-nil/nil value? What about errors?

The lack of predictability here is bad for whoever is reading this code, as well as people who will modify it in the future.

Design Better Pipelines in Elixir

As previously stated, to achieve predictability, we need three things in our code: structure, size, and simplicity. Knowing that, we can create a few rules to help us write predictable code:

Each function on a pipeline is expected to transform a given state into an updated state, and it should only do one transformation. This will help us keep functions small and focused.
Always receive two parameters. The first parameter is the state we want to transform, and the second contains any optional or extra data we might need to perform such a transformation. The second parameter is optional.
Always return an ok/error tuple. If everything goes well, an updated value of a state is returned.

For now, we won't require a structure or type for the error description.

Let's revisit our checkout code, now applying these rules:

options = %{conn: conn}

with {:ok, payment} <- fetch_payment_information(params, options),
     {:ok, user} <- fetch_user(conn),
     {:ok, address} <- fetch_address(%{user: user, params: params}, options),
     {:ok, order} <- create_order(%{user: user, address: address, payment: payment}, options)
  do
  conn
  |> put_flash(:info, "Order completed!")
  |> redirect(to: Routes.order_path(conn, order))
else
  {:error, error_description} ->
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end

It definitely looks better, and if someone in the future has to add a new step, they will know how this new function should look. Because the errors always have the same shape, we just need to find the appropriate message for a given error description, and everything else will look the same.

However, although we've solved the design of one particular feature, none of these changes or design principles are enforced. A team working on a different part of your product might never have contact with this code and may follow another design or pattern — or none at all.

If we can find a way to enforce these changes, then the code will always look the same.

An Emerging Pattern

Now we can see a pattern emerging when we write our functions: chain-of-state transformations — a pipeline. It's almost like an Enum.reduce/3 function.

However, instead of applying the same function to a collection of values, we apply a collection of functions to transform a state.

We'll explore this pattern in greater detail in the next part of this series.

Coming Up Next Time: Enforcing Predictable Elixir Code

In this article, we showed how to write code in a pattern that our brains can understand faster and more accurately. And by doing so, we also can write — and keep writing! — code that is maintainable in the long term.

Note that this pattern does not cover module organization, project structure, or API design. For that, I highly recommend the series of articles Towards Maintainable Elixir by Saša Jurić.

In the next and final part of this two-part series, we'll learn how to create a basic framework to enforce design principles in our Elixir code.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

LiveView Assigns: Three Common Pitfalls and Their Solutions

Roy Tomeij — 2022-06-28T00:00:00+00:00

In the first part of this two-part series, we examined LiveView assigns in detail — demystifying assigns, looking at some key concepts, and debugging.

Now, we'll turn our attention to three common mistakes that you might make with assigns and how to avoid them.

Let's get started!

1. Evaluating All LiveView Assigns

As you pass assigns around to view helpers, and the complexity increases, you may need many assigns in some functions. For example:

<%= user_note(@user, @note, @theme, @locale) %>

Then you may be tempted to do the following instead:

<%= user_note(assigns) %>

The problem with this simplification is that it’ll completely ruin change tracking and, as a result, changing any assign will trigger an update.

To solve this, stick to passing only the required assigns explicitly and collapse multiple arguments into a keyword list if needed:

<%= user_note(@user, @note, theme: @theme, locale: @locale) %>

Note: LiveView itself hints at, and counters this problem by excluding all other assigns from the widely used @socket struct in which they’re generally stored. But it does not forbid you from reaching for the assigns directly, opening a door to this issue.

2. Re-rendering Entire Lists

Change tracking on nested data such as lists is a complex problem, regardless of the framework. LiveView goes the extra mile to represent for loops via a dedicated struct so that static parts are only sent once. But when it comes to assigns, it tracks all that appear in such loops as a whole.

Our generated live resource in the 'Caveman Debugging in LiveView' section of the first part of this series re-evaluated every table row and cell regardless of what we did with the @notes assign.

There’s a solution though. We can establish a separate tracking context using stateful live components. So let’s try it out and create a component for each note:

defmodule MyAppWeb.NotesLive.Index.NoteRow do
  use MyAppWeb, :live_component
  def render(assigns) do
    ~H"""
    <tr id={"note-#{@note.id}"}>
      <td><%= inspect(Time.utc_now()) %></td>
      <td><%= inspect({Time.utc_now(), @note.name}) %></td>
      <td><%= inspect({Time.utc_now(), @note.content}) %></td>
      <!-- ACTIONS (CUT) -->
    </tr>
    """
  end
end

Then render it within the table:

<table>
  <!-- TABLE HEADER (CUT) -->
  <tbody id="notes">
    <%= for note <- @notes do %> <.live_component module={__MODULE__.NoteRow}
    id={"note-row-#{note.id}"} note={note} /> <% end %>
  </tbody>
</table>

Now you may once again create, edit, and delete some notes for the following highly desired behavior:

When creating, only the newly added row gets updated.
When editing, only the cells for changed fields get updated.
When deleting, no other rows get updated.

Note: You should not have to worry about memory usage caused by copying assigns to live components. They reside on the same process as the parent view, and so should share immutable data.

The excellent article Optimising data-over-the-wire in Phoenix LiveView compared Websocket payload for the naive loop vs. the component-based version. Note, however, that it may not reflect the current state of affairs as these things are rapidly evolving — see the Phoenix LiveView changelog.

3. Growing LiveView Assigns Infinitely

As noted in the 'LiveView Assigns Manage State' part of the previous post, the server-side nature of LiveView places extra responsibilities on memory management. For that reason, you can't afford to grow lists infinitely, e.g., when paginating, as shown below:

defmodule MyAppWeb.NotesLive do
  def render(assigns) do
    ~H"""
    <div id="notes">
      <%= for note <- @notes do %>
        <div id={"note-#{note.id}"}>
          <!-- CUT (note) -->
        </div>
      <% end %>
    </div>
    <button phx-click="load_more">Load more</button>
    """
  end
  def handle_event("load_more", _, socket) do
    next_page = socket.assigns.last_page + 1
    more_notes = Notes.list_notes(page: next_page)
    {:noreply, assign(socket,
      notes: socket.assigns.notes ++ more_notes,
      last_page: next_page
    )}
  end
end

Here, the memory usage will grow linearly, not just with the number of users (which is a problem we definitely don't want to have) but also with the total number of notes.

The solution is to mark that assign as temporary, switch to the append update model for the note listing, and only assign new items:

defmodule MyAppWeb.NotesLive do
  def mount(params, session, socket) do
    # ...
    {:ok, socket, temporary_assigns: [notes: []]}
  end
  def render(assigns) do
    ~H"""
    <div id="notes" phx-update="append">
      <!-- CUT (notes loop) -->
    </div>
    """
  end
  def handle_event("load_more", _, socket) do
    next_page = socket.assigns.last_page + 1
    more_notes = Notes.list_notes(page: next_page)
    {:noreply, assign(socket,
      notes: more_notes,
      last_page: next_page
    )}
  end
end

Now the notes list will still accumulate newly loaded records while keeping memory usage under control.

Wrap Up

In the first part of this series, we spent some time diving into LiveView assigns. First, we demystified assigns, before embarking on some caveman debugging and socket inspection.

In this article, we examined three common pitfalls — evaluating all assigns, re-rendering entire lists, and growing assigns infinitely — and their solutions.

I hope that this series has given you an idea of some trade-offs that come with LiveView assigns, alongside ways to get the most out of the wonderful technology that Phoenix LiveView definitely is.

Happy assigning!

A note from AppSignal: We're very excited to have recently released AppSignal for Phoenix 2.1, which adds automatic instrumentation for LiveView through Telemetry.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Guide to Phoenix LiveView Assigns

Roy Tomeij — 2022-06-14T00:00:00+00:00

Phoenix LiveView lets you develop full-stack apps with client-side interactions while mostly avoiding cross-stack hassle. Assigns, managed by the LiveView socket, are a core tool for making that happen — allowing you to store, present, and update data effortlessly and efficiently. But as they do so much, assigns come with their own complexities and may backfire if misused.

Over the last three years, I've had the pleasure to work on multiple LiveView apps within multiple teams, so I know first-hand that assigns can feel like magic to many developers (myself included).

In this article, we're going to demystify LiveView assigns. We'll describe what LiveView assigns actually are in relation to popular frameworks, find out how assigns work in practice, and highlight some of the traps that one may easily fall into.

Note: Throughout this post, we're going to use assigns to refer to LiveView assigns — as opposed to EEx assigns, Plug assigns, or Phoenix Socket assigns.

Demystifying Phoenix LiveView Assigns

Let's start with an overview of what assigns actually are. Here's a basic example:

defmodule MyAppWeb.StatsLive do
  use MyAppWeb, :live_view
  alias MyApp.{Accounts, Notes}

  def mount(params, session, socket) do
    {:ok, assign(socket,
      note_count: Notes.get_note_count(),
      user_count: Accounts.get_user_count()
    )}
  end

  def render(assigns) do
    ~H"""
    Note count: <%= @note_count %>
    User count: <%= @user_count %>
    """
  end
end

The above snippet shows how @note_count and @user_count are assigned initial values and then rendered as dynamic fragments among the static HTML that surrounds them. That's the concept of LiveView assigns in a nutshell.

Now let's take a deeper look, pointing out some similarities and differences between assigns and mainstream front-end development.

Let's get going!

LiveView Assigns Provide Dynamic Input

Assigns in LiveView build on top of those used by EEx as well as from "traditional" Phoenix. They are just input variables — placeholders for dynamic data provided from the outside.

Run the following code from iex:

iex> EEx.eval_string("Hello, <%= @name %>!", assigns: [name: "John"])
"Hello, John!"

We can easily compare assigns to props from React, Vue, or any other web component library. This is especially true for stateless components that have recently switched to simple function syntax, like functional components in React. Consider this simple component:

defmodule MyAppWeb.MyComponents do
  def my_button(assigns = %{text: _, click: _}) do
    ~H"""
    <button class="some-class" phx-click={@click}>
      <%= @text %>
    </button>
    """
  end
end

You can surely imagine yourself rewriting it in any web component library with a very similar code structure and volume. For the record, here's how similar it'd be in React:

function MyButton({ text, click }) {
  return (
    <button class="some-class" onClick={click}>
      {text}
    </button>
  );
}

The component uses assigned input wherever it needs to, while the caller of my_button provides it like this:

<.my_button text="Click me!" click="some_event" />

Easy enough, and no room for dilemmas. But that's just the start.

LiveView Assigns Manage State

For live views and live components, assigns grow from being a dumb input into a long-lived state. As such, they derive from Phoenix Channel assigns, and shift from props into state in the React nomenclature.

Views and components may opt to update their assigns from the inside to evolve their state. This is nothing fancy on its own, so — just like with props — we can easily find an equivalent in any web component framework.

What is fancy, however, is that in LiveView this happens on the server — taking advantage of Erlang processes to deliver low latency at scale.

To understand why this is great, let's take a look at the following example:

defmodule MyAppWeb.UserSubscriptionLive do
  use MyAppWeb, :live_view

  # ...

  def handle_event("pick_subscription", params, socket) do
    # encrypted session
    %{assigns: %{current_user: user}} = socket

    # CSRF-protected user input
    %{"plan" => plan} = params

    # ACID-safe database
    subscription = Subscriptions.create_subscription(user, plan)

    # non-public, secure APIs
    Payments.charge_user(user, subscription)

    # server-only background jobs
    Mailing.send_subscription_created_email(user, subscription)

    {:noreply, assign(socket, subscription: subscription})
  end
end

Without any effort, we mix data from the session, user input, databases, APIs, background jobs, and PubSubs — in one place, untampered, and server-side rendered. Presenting the results is just as simple:

<%= unless @subscription do %>
  Pick your plan:

  <a href="#" phx-click="pick_subscription" phx-value-plan="basic">Basic ($4.99)</a>
  <a href="#" phx-click="pick_subscription" phx-value-plan="premium">Premium ($19.99)</a>
<% else %>
  You're currently subscribed to <%= format_plan(@subscription.plan) %> plan.

  <%= if @subscription.last_payment do %>
    You were last charged on <%= DateTime.to_date(@subscription.last_payment.inserted_at) %>.
  <% end %>
<% end %>

Imagine all the moving parts that it would take to produce and fully test the client-side equivalent of our @subscription assign. This is perhaps the biggest highlight of LiveView, stealing the spotlight from API layers, state managers, sagas, and browser-based test suites.

At the same time, state management is undoubtedly a challenging task, and keeping memory usage under control is one of our main concerns. This is especially true considering we're on the server, and all connected users will claim the memory needed to hold the assigns during their entire session.

We'll examine a specific use case along with its solution in a moment.

LiveView Assigns Fuel Change Tracking

If you think that assigns are "just" props and state, and the show is over, brace yourself.

You see, if you update a single React prop or piece of React state, the whole component will re-render. But LiveView assigns are different. The reason for this is HEEx — a templating engine that splits your template into static and dynamic parts, and then only re-evaluates dynamic parts to involve the changed assigns.

Since this happens on the server, it's only the actual changes that will ever take up bandwidth, saving you the effort needed to slim down JSON payloads in classic SPAs, e.g., by designing case-specific or flexible APIs. That's some cool out-of-the-box optimization.

Note: This feature is known by different names, including change tracking, reactivity, memoization, or observability. The approach towards it varies across JS frameworks — check out the above keywords regarding Angular, Ember, Knockout, or Svelte.

In React, we can achieve similar behavior in a more explicit way (so it'll serve as a great visual exercise) with useMemo. Consider the following live component:

defmodule MyAppWeb.UserHoroscope do
  use MyAppWeb, :live_component

  def render(assigns) do
    ~H"""
    <div>
      <strong>Horoscope for {format_full_name(@first_name, @last_name)} (born on {format_date(@birthday)}):</strong>
      {generate_horoscope(@birthday)}
    </div>
    """
  end
end

Out of the box, it will only call format_date or get_horoscope when @birthday changes, but not when either @first_name or @last_name do (which may be extra useful if our generator that powers the horoscope becomes complex and the user name changes). The same goes for the other <%= ... %> snippets.

With React, we start off with the following naive component that runs all helper functions, regardless of changes:

function UserHoroscope({ firstName, lastName, birthday }) {
  return (
    <div>
      <strong>
        Horoscope for {formatFullName(firstName, lastName)} (born on{" "}
        {formatDate(birthday)}):
      </strong>
      {getHoroscope(birthday)}
    </div>
  );
}

Then, we'd extend it into the following shape with useMemo so that each helper is called only when its dependencies change:

function UserHoroscope({ firstName, lastName, birthday }) {
  const fullName = useMemo(() => {
    return formatFullName(firstName, lastName);
  }, [firstName, lastName]);

  const formattedBirthday = useMemo(() => {
    return formatDate(birthday);
  }, [birthday]);

  const horoscope = useMemo(() => {
    return generateHoroscope(birthday);
  }, [birthday]);

  return (
    <div>
      <strong>
        Horoscope for {fullName} (born on {formattedBirthday}):
      </strong>
      {horoscope}
    </div>
  );
}

Note: We're not saying React is the worst here. React focuses on optimizing historically slow DOM patching while assuming that modern JS is fast enough to re-evaluate typical rendering logic (ideal for a client-side library). LiveView goes the extra mile to save on server resources and bandwidth without sacrificing the dev experience — which works well for a compiler-backed server-side library.

Overall, change tracking is a powerful feature that does some really useful stuff under the hood while keeping our code clean. But it may also raise some ambiguity and doubts as far as assigns are concerned, and, if used without care, change tracking can definitely backfire performance-wise.

Taking LiveView Assigns for a Spin

Now that we know multiple concepts are mixed in with assigns, we see how they may actually not be as simple as they look in the code.

LiveView docs have a dedicated assigns and HEEx guide, so that's the first place you should look for answers and pointers. The module page for the template engine gives some extra insight.

But, sooner or later (and I'd put my bets on sooner considering the early stage of LiveView), that isn't enough. Let's look at how to approach LiveView to:

Play with it and boost your confidence
Debug and fix suspicious behavior in a real app
Check for optimizations or regressions in new LiveView releases

Caveman Debugging in LiveView

The first technique that you may want to employ involves returning to the very roots of debugging. You'll use a battle-proven, highly sophisticated technique that's been hiding in the darkest corners of the internet under numerous inspiring names like 'caveman debugging'.

The idea is to rely on the fact that HEEx only re-evaluates a subset of code blocks and to output a timestamp as close as possible to the place you want to inspect in the template.

To try it out, generate a sample live resource for testing in your Phoenix 1.6 app:

mix phx.gen.live Notes Note notes name:string content:text

Apply the printed instructions and navigate to /notes. Then add the following inspect/1 calls to your index.html.heex:

<%= inspect({Time.utc_now(), :above_notes}) %>
<table>
  <!-- CUT (table header) -->
  <tbody id="notes">
    <%= for note <- @notes do %>
      <tr id={"note-#{note.id}"}>
        <td><%= inspect(Time.utc_now()) %></td>
        <td><%= inspect({Time.utc_now(), note.name}) %></td>
        <td><%= inspect({Time.utc_now(), note.content}) %></td>
        <!-- CUT (actions cell) -->
      </tr>
    <% end %>
  </tbody>
</table>

Now try adding and removing some notes. In the default generated live resource, you should notice that the :above_notes timestamp updates together with all the per-row ones when creating or editing notes, but not when deleting them.

This hints that something different is happening with the live view and its assigns for the delete case. And rightfully so. When you look at the code, FormComponent uses push_redirect/2 upon a save, causing the whole index to reload, while Index uses assign/2,3 upon deletion to only update that specific assign.

If you want to avoid the reload, it's as simple as replacing push_redirect/2 in FormComponent with push_patch/2, plus reloading the @notes assign in Index like this:

defp apply_action(socket, :index, _params) do
  socket
  |> assign(:page_title, "Listing Notes")
  |> assign(:notes, list_notes())
  |> assign(:note, nil)
end

Voilà: your first performance tweak!

You may also have noticed that all per-row timestamps re-render when you create, edit, and delete notes — regardless of whether a specific note is affected or not. We'll get back to this later.

Socket Inspection

Sometimes, it's useful to see what LiveView pushes down the wire. This isn't so easy to visually tie to specific places on the page. But it allows us to inspect what is getting updated, and when, without template changes, and — this is especially useful — to see payload size and structure.

In order to inspect a socket:

Visit your live view in Chrome.
Open Developer Tools.
Switch to the Network tab.
Optionally filter by the WS type.
Choose the websocket?_csrf_token=... name.
Switch to the Messages tab.
Choose the message that you're interested in.

Here's how it might look:

When working with two notes, the highlighted piece of inbound payload is responsible for filling the listing. This confirms that the entire listing is re-rendered, regardless of operation.

Note: You may find it more handy to ask LiveView's JS client to log updates to the console, by calling liveSocket.enableDebug() in it.

In the end, having insight into the raw payload can lead you to some case-specific optimizations, such as:

Restructuring the assigns (e.g., by breaking apart large or nested ones)
Using live components (to ensure some assigns are tracked separately)
Sending some data to JavaScript hooks as binary (by switching to Phoenix.Channel)

Debugging in Production with AppSignal

The caveman technique is great for a single developer working on an application that hasn't been pushed to production. However, if you have an app in production with live users, you may want to take a look at AppSignal for monitoring your application performance and checking for errors in production. Adding AppSignal to an existing application takes a few seconds, and you can track LiveView errors from there.

Here's an example of the AppSignal dashboard collecting data from a sample application:

Wrap Up

In this post, we started off by demystifying LiveView assigns, before touching on some cross-over between assigns and mainstream front-end development. Finally, we explored caveman debugging in LiveView and socket inspection.

LiveView combines a lot of functionality in assigns, including some that's just as clever and useful as it is implicit and complex.

The key is to get a grasp on the key concepts, which are all relatable to what we can find in client-side frameworks. And, considering the framework's young age, to have a fallback solution for when existing resources can't help — a way to dig into it on your own.

Next up in this two-part series, we'll look at some common pitfalls when it comes to LiveView assigns.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Algebraic Data Types in Elixir

Roy Tomeij — 2022-05-31T00:00:00+00:00

Elixir is a dynamically-typed language. Types in Elixir are checked when a program runs, not when it compiles. If they don’t match up, an exception is thrown.

In statically-typed languages, types are checked during compile time. This can help us write code that is correct, understandable, and refactorable.

But it also introduces a certain focus on types as the foundation for your application. One interesting concept is to use types to model your business domain. In languages like Haskell, F#, and OCaml, this is usually done with algebraic data types (ADTs) — they build compound data types by aggregating types with product (AND) and sum (OR) types.

With the help of Dialyzer, a static analysis tool, you can use ADTs to constrain the number of your application's allowed states. This decreases the chance that errors will slip in.

In this article, we'll cover Dialyzer, ADTs, and how you can solve the problem of illegal states by using them together.

Let's get going!

Type Declarations in Elixir with Dialyzer

In Elixir (and other BEAM languages), checking type specifications is usually done with Dialyzer.

Dialyzer is different from the type systems of Haskell, OCaml, or even TypeScript. Instead of you proving to the compiler that your code is correct, Dialyzer needs to prove to you that your code is not correct.

This makes Dialyzer rather lax in requirements. If there is a way for the types to work, Dialyzer will assume you know what you are doing, and the types will, indeed, work.

But it’s still useful to reason about types and catch the occasional mistake.

Quick Intro to Dialyzer in Elixir

You can use Dialyzer in Elixir by adding a type spec for your functions via @spec.

#          (1)      (2)           (3)
  @spec plus_one(integer) :: integer

  def plus_one(x), do: x + 1
end

Here, we just write the name of the function (1) with its type as an argument (2), then the return type of the function (3).

You can find a list of basic types to use in Elixir's documentation.

You can also create your own type aliases via @type. To do that, you need to provide the alias' name (1) and its type (2).

#         (1)        (2)
  @type counter :: integer

If you use the Elixir language server, that’s all you need to do: your plugin/extension will notify you about any specs that Dialyzer doesn’t like. Otherwise, you need to run the mix dialyzer task to check your types.

You can find more about the use of Dialyzer on Elixir's website and in Elixir's typespec docs.

Now that we can type spec our Elixir code, let’s dig into algebraic data types.

Algebraic Data Types

While the name sounds scary (ooh, algebra 👻), algebraic data types are relatively simple.

This section will focus on the two main parts of ADTs: product (AND) and sum (OR) types.

Product Types

Product types are all around us. A product type is just a type with two or more fields that each hold a data type. You can also think of them as AND types.

For example, a tuple is a product type:

@type tuple(a, b) :: {a, b}

It takes two data types — a and b — and returns a type that holds both of these data types.

In Elixir, we also use product types with named fields — structs.

defmodule Person do
  defstruct first_name: "Gints", last_name: "Dreimanis"
end

Let's look at the type in this example:

@type t() :: %__MODULE__{
        first_name: String.t(),
        last_name: String.t()
      }

Generally, you can think of types as being sets of possible values. For example, a boolean has two possible values: :true and :false. A type for a traffic light color has one of three possible values: :green, :yellow, and :red.

If we have two types with sizes a and b, then the product type of those types will contain a * b values — the product of the sizes of those types. If you make a product type of the boolean and traffic light types — e.g., put a boolean and traffic light in a tuple — you'll have 2 * 3 = 6 possible values.

{:green, :true}
{:yellow, :true}
{:red, :true}
{:green, :false}
{:yellow, :false}
{:red, :false}

Sum Types

A historically less common kind of type is sum types.

In contrast to a product type, a sum type gives you one of two (or more) options. You can also think of them as OR types.

We use these in Elixir as well. For example, result tuples are sum types.

@type result(a, b) :: {:error, a} | {:ok, b}

In a result tuple, we can have an error of type a or success of type b.

Or, for example, we can have a sum type for optional values.

@type optional(a) :: :error | {:ok, a}

But it can also just be used for making lists of alternatives.

@type direction :: :north | :west | :south | :east

If you list two types in a sum type, the resulting type can pick a type from one set of values or the other. Because of this, its size is generally the sum of the sizes of those types.

The above might not always be true when using Dialyzer: you can put together two sets that overlap. For the statement to be true, they need to be tagged with the set they come from — the result type we defined above is a prime example.

That’s what people usually mean by algebraic data types.

There's More to Algebraic Data Types

By putting together sums and products, we have something akin to the algebra that we know and love from our school days: multiplication, sums, and variables.

Of course, there is more to algebraic types than just this: there’s also recursion, exponentials, etc. If you want to delve deeper into the subject, check out how they look in languages like Haskell.

Here are two Haskell articles that you can read: one from Joel Burget on the algebra of ADTs and an ADTs in Haskell tutorial from yours truly.

How Algebraic Data Types Help in Domain Modeling

We Elixir programmers usually don’t think in terms of sum types. The primary tool for modeling a domain in Elixir is the struct, a product type with named fields.

While that's good enough for most things, sometimes it is beneficial to use sum types as well.

Let’s look at an example.

Custom Kanban Board

Let’s imagine we need to create a representation of a customized Kanban board issue.

Our issues can be in one of these states:

Searching for assignee: in this case, it should have neither an assignee nor a reviewer
Not started yet: in this and the following state, it should have an assignee but not a reviewer
In progress
In review: in this and the following state, it should have an assignee and a reviewer
Done

All issues also have names and descriptions.

At first, one might be tempted to use a simple struct.

defmodule Issue do
  defstruct name: "",
            description: "",
            state: :searching_for_assignee
            assignee: nil,
            reviewer: nil
end

But as we saw in the section on product types, a simple product type has a lot of possible values, some of which might not match our requirements.

For example, we can create an issue that searches for an assignee but still has an assignee and a reviewer.

iex(1)> %Issue{name: "wrong issue", description: "not good at all", state: :searching_for_assignee, assignee: "Jorge Luis Borges", reviewer: "Gabriel García Márquez"}
%Issue{
  assignee: "Jorge Luis Borges",
  description: "not good at all",
  name: "wrong issue",
  reviewer: "Gabriel García Márquez",
  state: :searching_for_assignee
}

While it is possible to avoid doing that generally, it is simpler to make it impossible. We have a saying for this: “make illegal states unrepresentable”.

To do that, we need to create a sum type that covers all the states that we want to allow. It will enable us to cut out some of the wrong states by substituting products of values with sums of values.

First, let’s combine the state, assignee, and reviewer fields into one field: state.

defstruct name: "",
          description: "",
          state: :searching_for_assignee

After that, let’s define a sum type for state that will contain our specified options.

Let’s look at them again. Our issue can be in one of these states:

Searching for assignee
Not started yet but with an assignee
In progress and with an assignee
In review with an assignee and reviewer
Done, with the assignee and reviewer left for the historical record

It’s quite easy to define a type that reads almost like this:

@type state ::
        :searching_for_assignee
        | {:not_started, String.t()}
        | {:in_progress, String.t()}
        | {:in_review, String.t(), String.t()}
        | {:done, String.t(), String.t()}

So that it's simpler to understand, we can create aliases for the assignee and reviewer.

@type assignee :: String.t()
@type reviewer :: String.t()

Now, the type looks exactly like our list of rules.

@type state ::
        :searching_for_assignee
        | {:not_started, assignee}
        | {:in_progress, assignee}
        | {:in_review, assignee, reviewer}
        | {:done, assignee, reviewer}

All that's left is to create a type specification for the module struct (Issue) by using our state type.

@type t() :: %__MODULE__{
        name: String.t(),
        description: String.t(),
        state: state
      }

Here’s the complete module code:

defmodule Issue do
  defstruct name: "",
            description: "",
            state: :searching_for_assignee

  @type assignee :: String.t()
  @type reviewer :: String.t()
  @type state ::
          :searching_for_assignee
          | {:not_started, assignee}
          | {:in_progress, assignee}
          | {:in_review, assignee, reviewer}
          | {:done, assignee, reviewer}

  @type t() :: %__MODULE__{
          name: String.t(),
          description: String.t(),
          state: state
        }
end

Now we can test whether this type specification can stop us from making a logic error.

We’ll create a function that adds a reviewer to the issue, but we’ll slip a bug inside: it will not change the state of the issue. We’ll also add a type spec.

@spec add_assignee(Issue.t(), assignee) :: Issue.t()
def add_assignee(%{state: :searching_for_assignee} = issue, assignee_name) do
  %{issue | state: {:searching_for_assignee, assignee_name}}
end

Dialyzer will correctly return a type error here:

lib/issue.ex:21:invalid_contract
The @spec for the function does not match the success typing of the function.

Function:
Issue.add_assignee/2

Success typing:
@spec add_assignee(%{:state => :searching_for_assignee, _ => _}, _) :: %{
  :state => {:searching_for_assignee, _},
  _ => _
}

It’s a bit cryptic, but it basically means that Issue.add_assignee didn’t compile, and we should look into it! 🙃

As you can see, algebraic data types have saved us from making an error. Turns out, they’re not really a scary monster but a friend.

Benefits of Algebraic Data Types for Your Elixir App

Adopting algebraic data types for your Elixir applications is a two-step decision process.

The first step is to choose to use Dialyzer and typespecs. Dialyzer gives most of the benefits that any language with static typing would:

It's easier to catch errors you've made in code.
Types give extra information about code: what it does and what values it operates. This is helpful when trying to understand the code.
Once you have written your code, types can ensure the code still does the same thing (similar to tests) so it is easier to refactor.

Once you've adopted Dialyzer for your codebase, thinking in terms of algebraic data types should come naturally, and give a few benefits:

As we have seen in the example, sum types in particular let you cut down the possible states and make illegal states unrepresentable.
Having AND and OR in your vocabulary helps you build compound types in a way that's intuitive and understandable even to non-developers (domain experts).

Of course, ADTs are just one tool for software correctness — definitely not a panacea. But in general, ADTs are a helpful concept for anyone working with an Elixir codebase that uses Dialyzer.

If your codebase doesn't use Dialyzer, your first goal should be to introduce it, which is a much larger undertaking than changing how you do types when writing type specifications. That undertaking is unfortunately out of the scope of this article.

If you want to learn more about how ADTs can make your code clearer and help you ship better code, I invite you to check out Domain Modeling Made Functional by Scott Wlaschin or his blog post series called Designing with Types.

ADTs in Elixir: Wrap Up and Further Reading

In this post, we explored how to check type specifications with Dialyzer, before focusing on the two main parts of ADTs: product (AND) and sum (OR) types.

We then turned to the use of ADTs in domain modeling and finally touched on the benefits of ADTs.

If you want to explore algebraic data types, the best option is to get accustomed to a statically-typed functional language such as Haskell, OCaml, or F#.

Each of those languages has excellent materials for using types as the foundation of your code. The resource that would vibe the best with Elixir developers is probably the aforementioned Domain Modeling Made Functional since it also covers domain-driven design, which can confuse new Phoenix developers.

Elixir also has a library that emulates algebraic data types called Algae, which we used in one of our previous articles — Functional Programming in Elixir with Witchcraft. In general, Algae provides better tools than Dialyzer to build up a tower of abstractions from simple building blocks. Brave readers might find it rewarding to experiment with.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Livebook for Elixir: Just What the Docs Ordered

Roy Tomeij — 2022-05-24T00:00:00+00:00

While initially conceived as a tool for data exploration (much like Jupyter for Python), Livebook has deservedly become a sensation in the Elixir community.

It has been fantastic to see all the wonderful ways teams are leveraging Livebook for a range of different use cases. We have seen Livebooks being used to:

Create interactive documentation for libraries.
Build onboarding material and guides.
Audit and explore potential dependencies in your app.

Livebooks have also been used as the default REPL interface for project development.

In this post, we'll show how you can easily create interactive documentation with Livebook and outline some top tips for using Livebook. We will assume you have installed Livebook, following the guidance in their README.

But First: What is Livebook for Elixir?

Livebooks are supercharged markdown files where you can add sections of arbitrary executable Elixir code. They are inspired by similar notebooks for other languages (like Python's Jupyter), but Livebooks leverage LiveView and other BEAM goodies, so they are even better.

Livebook files get their own .livemd extension, and (somewhat confusingly) we create and run those Livebook markdown files using a Phoenix application also called Livebook.

That phoenix app runs in the browser and enables a whole host of interactive features like collaboration, as we will see.

The expectation is that you will install the Livebook repo locally and start a Livebook server from somewhere on your machine where there are Livebooks (the files).

The Livebook app will show you the working directory of where you started the Livebook server, so you can select any given Livebook to run from there.

Let's now look at a library to document.

Livebook Docs: A Single Source of Truth in Elixir

Our library is going to accept various inputs and rate them according to the following chonk chart:

It will output the chonk rating accordingly. First, let's create the library.

mix new chonk_o_meter && cd chonk_o_meter

Let's add ex_doc to our deps in our mix.exs:

defp deps do
  [
    {:ex_doc, ">= 0.0.0", runtime: false, only: [:docs, :dev]},
  ]
end

Now, download the chonk chart image above and put it into the root of the library in a directory called images so we can refer to it in our README. In the README.md, let's put a title and a short explanation of what the library aims to do:

# Chonk 'O' Meter

Chonk O Meter is a state-of-the-art size estimator. It will rate the size of anything according to the following chart:

![alt chart showing cats of various sizes](./images/chonk.jpg)

So far, so good. Now we will open the module and write our moduledoc. But here's the thing: what we really want to do is just copy what we already wrote for the README.

Having one source of truth for that information is super valuable as the library develops because having to update the documentation in multiple places is a recipe for errors!

It's good to repeat the same information in different formats because different people will come to the library via different paths.

Some may see the repo first (and therefore the README), whereas others may see the library on Hex first — and so only see the moduledoc.

You may think that it's easy enough to copy and paste, but once we add our Livebook into the mix, there will be three places we need to update docs when something changes!

Instead, let's be a bit smarter. We will section off a part of the README and read that section into the moduledoc at compile time. Sandwich the introduction to the library between two markdown comments in the README:

<!-- README START -->

.... Library introduction here.

<!-- README END -->

Now, we can write the introduction as if it were a moduledoc in between those two comments, like so:

<!-- README START -->

Chonk O Meter is a state-of-the-art size estimator. It will rate the size of anything according to the following chart:

![alt chart showing cats of various sizes](./images/chonk.jpg)

<!-- README END -->

In the main module ChonkOMeter, we can do this:

defmodule ChonkOMeter do
  @moduledoc File.read!(Path.expand("./README.md"))
             |> String.split("<!-- README START -->")
             |> Enum.at(1)
             |> String.split("<!-- README END -->")
             |> List.first()
end

This will extract the guide sandwiched between the markdown comments and set it as the moduledoc. Now we only need to change the README to update both!

We can generate the documentation and view it locally to verify that this works as expected. If you run mix docs, a doc folder will appear with an index.html file. We can open doc/index.html to view our documentation in the browser.

Adding an Image to the Moduledoc

If you navigate to the module's documentation, you will notice that the image is missing. Hex allows you to point to assets in your docs as long as they are included inside the doc directory (generated when you create docs with the mix docs command).

Usually, that command overwrites the whole doc folder. So, to ensure that our pictures are always copied there, we can use an alias in our mix.exs file. We will turn the mix docs command into one that runs mix docs and then copies all images inside the /images directory into a doc/images directory.

defmodule ChonkOMeter.MixProject do
  use Mix.Project

  def project do
    [
      ...
      aliases: aliases(),
      ...
    ]
  end

  defp aliases() do
    [docs: ["docs", &copy_pictures/1]]
  end

  defp copy_pictures(_) do
    File.cp_r(Path.expand("./images/"), Path.expand("./doc/images/"))
  end
end

If you run mix docs now, you will see that the images/ directory gets copied over to the doc folder. Open the doc/index.html file and you should now see the chonk chart appear!

Doctests in Elixir's Livebook

It's important to note that in writing our moduledoc in this way, we don't lose any of the usual capabilities ex_docs give us.

Anything you can normally do in a doctest, you can still do here. To demonstrate that, let's add a doctest to our README. First, we'll need a function to test:

defmodule ChonkOMeter do
  @moduledoc File.read!(Path.expand("./README.md"))
             |> String.split("<!-- README START -->")
             |> Enum.at(1)
             |> String.split("<!-- README END -->")

  @doc """
  Returns the Chonk rating for a given number of story points.
  """
  def story_points(points) when is_integer(points) and points >= 0 and points < 3 do
    "A Fine Boi"
  end

  def story_points(points) when is_integer(points) and points >= 3 and points < 5 do
    "He Chomnk"
  end

  def story_points(points) when is_integer(points) and points >= 5 and points < 8 do
    "A Heckin' Chonker"
  end

  def story_points(points) when is_integer(points) and points >= 8 and points < 10 do
    "H E F T Y C H O N K"
  end

  def story_points(points) when is_integer(points) and points >= 10 and points < 15 do
    "Mega Chonk"
  end

  def story_points(points) when is_integer(points) and points >= 15 do
    "Oh Lawd He Comin'"
  end
end

Now remove the boilerplate in our test file, so it looks like this:

defmodule ChonkOMeterTest do
  use ExUnit.Case
  doctest ChonkOMeter
end

Run the tests to ensure there are none for now:

mix test

Finally, in our README, we can add the usual doctest syntax:

<!-- README START -->

Chonk O Meter is a state-of-the-art size estimator. It will rate the size of anything according to the following chart:

![alt chart showing cats of various sizes](./images/chonk.jpg)

For example:

    iex> ChonkOMeter.story_points(10)
    "Mega Chonk"

<!-- README END -->

If you run the tests, you will notice that the change in the README has not triggered a re-compilation, meaning the app still thinks there are no doctests. To fix this, we just need to add an @external_resource module attribute into the main module. This tells mix to recompile when the README changes:

defmodule ChonkOMeter do
  @external_resource Path.expand("./README.md")
  # ^^ Add this line ^^
  @moduledoc File.read!(Path.expand("./README.md"))
             |> String.split("<!-- README START -->")
             |> Enum.at(1)
             |> String.split("<!-- README END -->")

  @doc """
  Returns the Chonk rating for a given number of story points.
  """
  def story_points(points) when is_integer(points) and points >= 0 and points < 3 do
    "A Fine Boi"
  end

  def story_points(points) when is_integer(points) and points >= 3 and points < 5 do
    "He Chomnk"
  end

  def story_points(points) when is_integer(points) and points >= 5 and points < 8 do
    "A Heckin' Chonker"
  end

  def story_points(points) when is_integer(points) and points >= 8 and points < 10 do
    "H E F T Y C H O N K"
  end

  def story_points(points) when is_integer(points) and points >= 10 and points < 15 do
    "Mega Chonk"
  end

  def story_points(points) when is_integer(points) and points >= 15 do
    "Oh Lawd He Comin'"
  end
end

When we run our tests, this now results in one passing doctest! We can also add a doctest to our function doc like so:

...
  @doc """
  Returns the Chonk rating for a given number of story points.

      iex> ChonkOMeter.story_points(5)
      "A Heckin' Chonker"
  """
  def story_points(points) when is_integer(points) and points >= 0 and points < 3 do
    "A Fine Boi"
  end

  def story_points(points) when is_integer(points) and points >= 3 and points < 5 do
    "He Chomnk"
  end

  def story_points(points) when is_integer(points) and points >= 5 and points < 8 do
    "A Heckin' Chonker"
  end

  def story_points(points) when is_integer(points) and points >= 8 and points < 10 do
    "H E F T Y C H O N K"
  end

  def story_points(points) when is_integer(points) and points >= 10 and points < 15 do
    "Mega Chonk"
  end

  def story_points(points) when is_integer(points) and points >= 15 do
    "Oh Lawd He Comin'"
  end
...

Adding Livebook to Elixir

So let's recap. Right now, we have a README as the source of truth for our moduledoc. We can have doctests and images and all the usual goodies that a moduledoc is allowed, but we don't have to repeat ourselves and risk copy/paste errors.

We want to keep that same energy going for our Livebook, to avoid repeating ourselves manually, but still have an interactive playground for our library on top of the usual moduledocs.

To do that, we can generate our Livebook from our module. To help with this, I've written a library we can include called livebook_helpers:

defp deps do
  [
    {:ex_doc, ">= 0.0.0", runtime: false, only: [:docs, :dev]},
    {:livebook_helpers, ">= 0.0.0", only: [:docs, :dev]},
  ]
end

Once we have fetched the deps with mix deps.get, running mix help shows one extra mix task:

...
mix create_livebook_from_module # Creates a livebook from the docs in the given module.
...

We can see from the docs that we run the mix task by providing a module and a path to a Livebook. Let's try that:

mix create_livebook_from_module ChonkOMeter "chonk_o_meter_introduction"

You should see a successful output that links to the generated Livebook! 🎉 There is one last thing we can do to make our workflow seamless. Let's add create_livebook_from_module to the end of the mix docs command.

defmodule ChonkOMeter.MixProject do
  use Mix.Project

  def project do
    [
      ...
      aliases: aliases(),
      ...
    ]
  end

  defp aliases() do
    [docs: ["docs", &copy_pictures/1, &create_livebook/1]]
  end

  defp copy_pictures(_) do
    File.cp_r(Path.expand("./images/"), Path.expand("./doc/images/"))
  end

  defp create_livebook(_) do
    Mix.Task.run("create_livebook_from_module", ["ChonkOMeter", "chonk_o_meter_introduction"])
  end
end

Whenever we run mix docs, we will copy over any static images used in the README and generate a Livebook from our main module!

Running Livebook in Elixir

So far, so good! We have a nice pipeline to create a useful Livebook, but now we need to think about running the Livebook. Start the Livebook app like so:

livebook server

By default, the Elixir sections only have access to the Elixir and Erlang standard library. If we run our generated library and then attempt to run an Elixir cell that calls the library, it will fail because the library code is not there. To solve this, we have two options — Mix.install or Livebook runtime.

Add `Mix.install` to Livebook

We could add a section to the beginning of the Livebook that does this:

Mix.install([:chonk_o_meter])

When called, we will get the latest version of the library from Hex. It will be made available to all subsequent Elixir cells, just like when you run Mix.install inside an IEx REPL.

You can also easily specify a version and provide live documentation for any version of a given library:

Mix.install([{chonk_o_meter: ">=0.0.1"}])

LivebookHelpers can even generate a Livebook with a Mix.install at the beginning if we supply deps to the mix task:

mix create_livebook_from_module ChonkOMeter "chonk_o_meter_introduction" "[:chonk_o_meter"]"

This works great for any library that is deployed to Hex. However, you'll run into problems if, for example, you want a Livebook for a main branch. In that case, you can do this:

Mix.install [{:chonk_o_meter, path: "./"}]

This tells mix to look in the provided path for a local version of the library. This, of course, makes some assumptions about where the Livebook will run, so it's good to make that clear. If you put the Livebook at the root of the repo and a user starts the Livebook server from there, then the path "./" will work.

If you don't want to rely on this, though, Livebook has your back with a powerful feature: runtime!

Livebook Runtime

There are three kinds of runtime, but they all let you point to code and call it in any Elixir cell within your Livebook.

The three runtime options are:

Embedded
Attached node
Mix standalone

You select the runtime by clicking the cog symbol here:

Let's look at each runtime option below.

Embedded Mode

Embedded Mode lets us run the notebook code within the Livebook node itself! This is really for specific cases where there is no option to start a separate Elixir runtime — for example, on embedded devices.

Code defined in one notebook may interfere with code from another notebook. So this mode should only be used if you have no alternative and is not relevant here.

Attached Node

The attached node runtime connects a Livebook to a running Elixir app via the usual Erlang magic that we use to connect two running nodes. It looks like this:

As long as the app starts with a cookie that you know and a sname, you can give them to Livebook and connect. This is like getting a remote shell in a running app but with a more full-featured text-editing environment.

Using an attached node gives you complete control over how your app starts. You get much more control than the mix standalone and can do all sorts of things, like set env vars and start other services (like a database).

An attached node could be especially relevant for creating internal (live!) documentation for closed-source repos at work, but is not relevant for us and our library.

Mix Standalone

Finally, the Mix standalone runtime lets you point to a mix project which Livebook will compile and start (analogous to running iex -S mix in your terminal).

Mix standalone confers a great advantage over Mix.install, as you can recompile it! It's useful if you write a Livebook from scratch; in that case, you'll likely add something to the library that you'll then want to use in the Livebook.

Instead of having to kill the Livebook server and restart to access the new functions, you can add an Elixir cell with the following:

IEx.Helpers.recompile()

This will recompile the connected mix app (i.e., our library) when you call it, meaning you get access to any new functionality therein.

Livebook for Docs: Try It Yourself in Your Elixir App

That concludes our tour of Livebook for docs. You can see the chonk_o_meter library example here. For an example of these ideas in action in a real library, check out my data_schema library too.

Currently, GitHub doesn't recognize the .livemd extension, so if you play around with Livebook, I would encourage you to push to public repos. The more we do this, the more chance we have of getting GitHub to parse the files with nice syntax highlighting and markdown rendering.

Until that happens, though, we can put a magic line at the top of a Livebook to force GitHub to render it as markdown:

<!-- vim: syntax=markdown -->

The Livebook that Livebook helpers generates will include this line for you. Now you have all the knowledge you need go forth and create live docs!

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Guide to Event-Driven Architecture in Elixir

Roy Tomeij — 2022-05-10T00:00:00+00:00

In this post, we will explore how event-driven architecture can make your app more responsive for users and decouple your modules for a better developer experience.

But first: what is event-driven architecture, exactly?

Event-Driven Architecture: An Introduction

Event-driven architecture is an architecture where events control the behavior and flow of your application. The main components of the architecture are event producers, event bus, and event consumers.

An event could be anything that represents a change of state in the system. For example, in an e-commerce application, the purchase of a product by the user could produce a sold event which the consumer can then process to update inventory.

An event-based architecture allows applications to act on events as they occur. Different parts of an application work and develop relatively independently in well-crafted event-based design. Organizations can assign separate teams to focused parts of the application and streamline the workflow. This also creates a clear boundary between different parts of an application, assisting in future scalability exercises.

Event-driven architecture has mainly gained popularity with microservice-based products but can also be used for a monolith.

Things are always clearer with an example, so let's look at one.

Building Blocks of Event-Driven Architecture

Let's discuss each building block in detail, using an e-commerce application as an example.

Imagine that a user makes a new purchase on a website. The new order is an event generated by the part that controls the ordering: the event producer.

The event can be pushed onto an event bus. The event bus could be anything, such as:

A table in the database.
An in-memory event queue inside the app.
An external tool like RabbitMQ or Apache Kafka.

The event consumers interested in this type of event can subscribe to the event bus. The event is delivered to them, and they do some processing on top of it.

For example, an inventory management system would subscribe to the new order event and update the inventory of the product. Another system could also pick the same event in the application — for example, a fulfillment service might process that event and create a delivery route for the product.

Benefits of Event-Driven Architecture

There are several advantages of using an event-driven architecture instead of one that processes everything sequentially.

An event-driven architecture allows us to build several independent parts of an application to work off the same event and do different focused tasks. This can be advantageous for a couple of reasons:

One team needs to focus on only one part.
The application code for small parts can be simple.

Another advantage of the design is that it allows us to deliver a snappy interface to the user. As an example from the above application, a user needs to make an order. The application can continue processing all other non-interactive tasks, like updating its inventory and interacting with the delivery application, without the user's attention.

This also makes adding new processing steps in the event pipeline very easy. For example, let's say we need an additional task to be performed from an event. We only need to add a new consumer to handle the event, without touching any other parts of the application.

Finally, it is much easier to scale each individual module if it is decoupled from the others, than to scale a whole application together. This is even more beneficial when you have a part that takes much more resources than its counterparts in the event processing pipeline.

But event-driven architecture does not come without disadvantages when not properly thought out. If applied to very simple problems, it can lead to complex workflows that are slow and difficult to debug.

Let's explore some simple ways event-driven architecture can be implemented with Elixir, without the need to write complex pieces of code.

Synchronous Event-Driven Architecture in Elixir

The simplest (and most inefficient) way to run the above flow would be to do everything synchronously in the user request.

So, if you have an Orders module that processes a user's order request, the synchronous implementation could look like this:

defmodule Orders do
  def create_order(attrs) do
    {:ok, order} = save_order(attrs)
    {:ok, _inventory} = update_inventory(order)
    {:ok, _delivery} = create_delivery(order)
    {:ok, order}
  end
end

We can improve this to more easily scale for new event consumers:

defmodule Orders do
  @event_consumers [
    {Inventory, :handle_event},
    {Delivery, :handle_event},
  ]

  def create_order(attrs) do
    {:ok, order} = save_order(attrs)

    event = %Orders.Event{type: :new_order, payload: order}
    @event_consumers
    |> Enum.each(fn {module, func} ->
      apply(module, func, [event])
    end)

    {:ok, order}
  end
end

With this implementation, all we need to do to add new consumers is to add the specification in the @event_consumers array, and those consumers can work independently.

While the synchronous approach works well for a small number of consumers, it has a disadvantage. Creating an order might take a long time because you will need to wait for the inventory to update and the delivery to be created.

These are all internal tasks that can be performed without user interaction and moved from the synchronous chain.

Let's see how we can further streamline event-driven architecture using GenServer.

Using GenServer for Event-Driven Architecture in Elixir

For this implementation, we will run a separate process for each consumer. These will subscribe to an event from a producer and run their tasks concurrently.

For the event bus, we can use Phoenix.PubSub. Note that it is possible for apps that don't use Phoenix to directly use Registry as a PubSub.

First, let's look at the producer.

defmodule Orders do
  def create_order(attrs) do
    {:ok, order} = save_order(attrs)
    event = %Orders.Event{type: :new_order, payload: order}
    Phoenix.PubSub.broadcast(:my_app, "new_order", event)
    {:ok, order}
  end
end

On a new order, we create the event struct and use Phoenix.PubSub.broadcast/3 to broadcast that event on the bus. As you can see, it is much simpler than the previous implementation where the Orders module processed tasks from the other module serially.

The consumers can then subscribe to the new_order topic and implement the handle_info/2 to be notified every time a new event is published by the producer.

defmodule Inventory do
  use GenServer

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  def init(_opts) do
    Phoenix.PubSub.subscribe(:my_app, "new_order")
  end

  def handle_info(%Orders.Event{type: :new_order, payload: order}, state) do
    state = consume(state, order.product)
    {:noreply, state}
  end
end

The Delivery module will be very similar to the above, so I am skipping it here.

As you can see, this is much better than the previous implementations. Inventory and Delivery modules can independently subscribe to the new_order topic. The Orders module broadcasts to this topic on new orders and events are delivered to the subscribed processes.

You could even distribute this between multiple nodes and Phoenix.PubSub (with a PG, Redis, or other adapter), spreading the events to all nodes.

Great, right? Not really. There are several issues with this approach:

PubSub provides a real-time broadcast without message queueing, so if one of the subscriber processes is down, it might miss the broadcasts.
If the subscriber does some heavy work, it might not be able to keep up with the incoming messages, resulting in timeouts and consequently crashing the process tree.
If the subscriber experiences an error when processing a message, it is considered consumed and won't be retried later.

So this approach is a bad one to follow for our current use case.

However, the approach still has its use cases. It can be used for tasks that aren't critical, or that can be corrected with the next message: for example, if a task computes the suggestions for a user's next purchases based on their last purchase. While this would also need to be triggered on a new order, it isn't exactly critical (for a traditional e-commerce website) and can re-compute suggestions for a user on their next purchase.

Event-Driven Implementation Using GenStage in Elixir

In the previous section, we saw a great implementation of our event-driven system using GenServer. But it didn't come without its limitations. Let's see how GenStage fares.

GenStage makes a clear distinction between producers, consumers, and producer_consumers, and each process has to pick one when it starts (in its init/1). In our case, both Inventory and Deliver are consumers, and Orders is a producer.

This is where things start to get a little complicated. GenStage has a concept of demand. Each consumer can issue a demand of how many events it can handle. The producer needs to send those events to the consumer. Let's see a basic producer in action.

defmodule Orders do
  use GenStage

  def start_link(opts) do
    GenStage.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_opts) do
    {:producer, :some_state_which_does_not_currently_mattere}
  end

  def create_order(pid, attrs) do
    GenStage.cast(pid, {:create_order, attrs})
  end

  def handle_cast({:create_order, attrs}, state) do
    {:ok, order} = save_order(attrs)
    {:noreply, [%Orders.Event{type: new_order, payload: order}], state}
  end

  def handle_demand(_demand, state), do: {:noreply, [], state}
end

The meat of our code is in handle_cast, where we save the order and return a tuple like {:noreply, events, new_state}. The new events are stored in an internal GenStage buffer and dispatched to the consumers as they make new demands (or immediately, if there are consumers with unmet demand).

Let's check out a sample implementation of the consumer:

defmodule Inventory do
  use GenStage

  def start_link(opts) do
    GenStage.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_opts) do
    {:consumer, [], subscribe_to: [Orders]}
  end

  def handle_events(events, _from, state) do
    state = Enum.reduce(events, state, & handle_event(&1, &2))
    {:noreply, [], state}
  end

  def handle_event(%Orders.Event{type: :new_order, payload: order}, state) do
    new_state = update_inventory(order)
    new_state
  end
end

In the consumer, first notice that we have a subscribe_to inside init/1. This automatically subscribes Inventory to any events published by Orders. Please check the GenStage documentation for additional options available in init.

Here, most of the work happens inside handle_events/3, which is automatically called by GenStage as soon as new events become available. We handle the new_order event here, updating the inventory and returning a new state.

With this simple implementation, we get several benefits that outpace the GenServer implementation:

Automatic buffering of events inside GenStage's internal buffer when the producer has new events without any available consumer.

Even if consumers are down when some events are produced, we are still guaranteed to receive them when the consumer comes back up.

Check out Genstage's guide on buffering demand for advanced buffering logic.

Automatic distribution of work on multiple consumers.

If you have heavy consumer tasks, you can start multiple consumer processes. The default DemandDispatcher for GenStage will distribute the work evenly across all processes.

See GenStage.Dispatcher for other dispatch strategies to distribute events to all consumers or partition distribution to consumers based on a hash function.

But, as with GenServer or synchronous implementation, using GenStage does not come without its problems.

If a consumer crashes while it is processing an event, GenStage will consider the event delivered and will not send it again when the consumer comes back up.

To make sure that you properly track crashes, you can use a monitoring service like AppSignal. AppSignal is easy to install for your Elixir app and helps you monitor performance as well as track errors. Here's an example of an error tracking dashboard that AppSignal provides:

You can set up notifications for crashes via AppSignal as well.

On the app side, you can cache such events in a persistent store once they are delivered to the consumer. If the consumer crashes, then once it recovers, it can revert to the cached events.

Be very cautious about producing too many events without enough consumers, though. While GenStage offers automatic buffering of events, this buffer has a (configurable) maximum size and a practical maximum size constrained by the server's memory.

If you don't control the frequency with which events are produced, consider using an external data store like Redis or Postgres to buffer events.

Wrap Up: Event-Driven Architecture in Elixir — Going Beyond GenStage

In this post, we examined three approaches to implementing an event-driven system in Elixir: synchronously, using GenServer, and finally, using GenStage. We took a look at some of the advantages and disadvantages of each approach.

The simple GenStage example can be a starting point for implementing complex event-driven data processing pipelines that span multiple nodes. I suggest you read the great GenStage documentation for more information.

If you are looking for an even higher-level abstraction, Broadway is a good starting point. It is built on top of GenStage and offers several additional features, including consuming data from external queues like Amazon SQS, Apache Kafka, and RabbitMQ.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Using Profiling in Elixir to Improve Performance

Roy Tomeij — 2022-04-26T00:00:00+00:00

Elixir is all about performance. Say you have an app up and running with Elixir, but some parts aren't working as fast as you would like them to.

That is where profiling comes in. Profiling tools usually walk you through the frequency and duration of function calls and where they spend their time. Erlang has impressive profile tooling available at its disposal.

In this post, we will look into three profiling tools in Elixir: cprof, eprof, and fprof.

Do I Need Profiling for My Elixir App?

Firstly, I want to clarify that you may not need profiling at all. As Donald Knuth wrote: 'premature optimization is the root of all evil.' Some operations just cannot be optimized.

For example, say you want to fetch something from your database. You issue a query and get a reply. This incurs a network round trip with some data. If you need all the data, the network round trip time cannot be optimized without moving your database closer to the app.

With that out of the way, let's jump into profiling options.

Profiling with `cprof` in Elixir

cprof counts the number of times each function is invoked. It doesn't profile the time spent in those functions, so it comes with the least amount of overhead.

Use it when you already have an estimate of your functions' runtimes. You'll see the frequency with which each function is invoked.

Let's see how we can profile a Fibonacci number generator with cprof.

You can follow along with the code. It uses the Rosetta Code algorithm for Fibonacci sequence generation.

Jump into an IEx session and run cprof like this:

iex(15)> :cprof.start()
19244
iex(16)> FibonacciRosettaCode.list(10)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
iex(17)> :cprof.analyse(FibonacciRosettaCode)
{FibonacciRosettaCode, 79,
 [
   {{FibonacciRosettaCode, :fibonacci, 3}, 54},
   {{FibonacciRosettaCode, :fibonacci, 1}, 11},
   {{FibonacciRosettaCode, :"-fun.fibonacci/1-", 1}, 11},
   {{FibonacciRosettaCode, :module_info, 1}, 1},
   {{FibonacciRosettaCode, :list, 1}, 1},
   {{FibonacciRosettaCode, :__info__, 1}, 1}
 ]}
iex(18)> :cprof.stop()
19266

First, start cprof, run the code that you want to profile, and then call :cprof.analyse/1 to fetch the statistics for the given module. Several other options are available inside the :cprof module, like pausing profiling with :cprof.pause/0.

We can also use Mix's wrapper task profile.cprof directly.

➜ mix profile.cprof -e "FibonacciRosettaCode.list(10)"
                                                                     CNT
Total                                                                113
FibonacciRosettaCode                                                  77  <--
  FibonacciRosettaCode.fibonacci/3                                    54
  FibonacciRosettaCode.fibonacci/1                                    11
  FibonacciRosettaCode."-fun.fibonacci/1-"/1                          11
  FibonacciRosettaCode.list/1                                          1
Enum                                                                  24  <--
  Enum.reduce_range/5                                                 12
  anonymous fn/3 in Enum.map/2                                        11
  Enum.map/2                                                           1
  ...

Here, we can see that the function fibonacci/3 is called the most. So, if we can somehow decrease the number of calls to that function or optimize the code inside it, we could improve performance.

`eprof` Profiling in Elixir

cprof measures counts while eprof measures the execution time (in addition to counts) spent inside each function. It has slightly more overhead than cprof. Use eprof when you want to find the most time-consuming functions.

The usage is very similar to cprof. You just need to start the profiler, run the code you want to profile, and then call analyze to fetch the results.

Let's analyze the results from the mix task profile.eprof on our sample Fibonacci generator.

➜ mix profile.eprof -e "FibonacciRosettaCode.list(10)"
#                                               CALLS     % TIME µS/CALL
Total                                             106 100.0   45    0.42
anonymous fn/0 in :elixir_compiler_1.__FILE__/1     1  0.00    0    0.00
:lists.reverse/1                                    1  2.22    1    1.00
:lists.reverse/2                                    1  2.22    1    1.00
FibonacciRosettaCode.fibonacci/1                   11  4.44    2    0.18
Enum.map/2                                          1  4.44    2    2.00
Range.new/2                                         1  4.44    2    2.00
FibonacciRosettaCode."-fun.fibonacci/1-"/1         11  8.89    4    0.36
FibonacciRosettaCode.list/1                         1  8.89    4    4.00
:erlang.apply/2                                     1  8.89    4    4.00
anonymous fn/3 in Enum.map/2                       11 13.33    6    0.55
Enum.reduce_range/5                                12 13.33    6    0.50
FibonacciRosettaCode.fibonacci/3                   54 28.89   13    0.24

Here, we can see that fibonacci/3 is again the most time-consuming part of our program, consuming 28.89% of the total execution time and 0.24µS per call.

In addition, eprof also allows us to profile function calls across different processes. The usage is very simple. Let's see an example:

:eprof.start_profiling([self()])

# Do some work
1..100 |> Enum.each(fn i ->
  spawn(fn -> FibonacciRosettaCode.list(i + 1) end)
end)

:eprof.stop_profiling()
:eprof.analyze()

You need to call start_profiling/1 with a list of processes to profile. By default, this also tracks any other processes started from a profiled process.

When you are done, call stop_profiling and run analyze to get the results. They should look something like this, with an entry for each process and the percentage of time it was busy:

****** Process <0.150.0>    -- 37.26 % of profiled time ***
FUNCTION                                                             CALLS        %   TIME  [uS / CALLS]
--------                                                             -----  -------   ----  [----------]
io:request/2                                                             3     0.00      0  [      0.00]
...

****** Process <0.277.0>    -- 0.05 % of profiled time ***
FUNCTION                                             CALLS        %  TIME  [uS / CALLS]
--------                                             -----  -------  ----  [----------]
'Elixir.FibonacciRosettaCode':fibonacci/3                2     0.00     0  [      0.00]
'Elixir.FibonacciRosettaCode':fibonacci/1                3     3.03     1  [      0.33]
'Elixir.FibonacciRosettaCode':'-fun.fibonacci/1-'/1      3     3.03     1  [      0.33]
...

The great thing about this is that you can run it in production systems by accessing the remote console. Just call start_profiling/1, wait some time to process the requests or manually trigger some requests that you want to profile. Then call stop_profiling followed by analyze to get the results.

Using `fprof` in Elixir

The final inbuilt tool that you can use to profile your applications is fprof. It is a comprehensive profiling tool that generates a trace file containing timestamped entries for function calls, process-related events, and garbage collection data.

You can then feed this trace file into other tools to visualize the results thoroughly or use :fprof.analyse like above to fetch function counts and execution times.

Running fprof through IEx is quite advanced compared to running cprof and eprof, but fprof allows tracing multiple processes.

Here is how you can generate a trace file for all processes:

:fprof.start()
:fprof.trace([:start, procs: :all])

# Do some work
1..100 |> Enum.each(fn i ->
  spawn(fn -> FibonacciRosettaCode.list(i + 1) end)
end)

:fprof.trace(:stop)
:fprof.profile()
:fprof.analyse(totals: false, dest: 'prof.analysis')

This generates a comprehensive trace of everything that went on during the run with:

CNT - the number of times a function is called
ACC - the accumulated time spent in the function, including other function calls
OWN - the time the function spent to be executed

Here's the full file (50 MB), if you are interested.

Like eprof, fprof is a great tool for profiling applications in production directly. Note that this will significantly slow down the application, so be prepared for degraded performance during profiling and a huge trace file.

So you have the data now. If you are feeling adventurous, you can start digging through the file and look for patterns.

Or, you can use some tools to help visualize your data. A popular one is erlgrind. It converts the fprof file to cgrind format that you can open inside KCachegrind.

To convert the file, download erlgrind and then run src/erlgrind profile.fprof. This will generate a prof.cgrind file that you can open to see graphs like this:

Profile and Collect Metrics from Your Elixir App in Production

eprof and fprof can assist with profiling your application in production, but there are a couple of additional tools worth mentioning — Phoenix Live Dashboard and AppSignal.

Phoenix Live Dashboard

Phoenix's live dashboard can provide a great, quick overview of metrics. While not exactly a profiler, it shows OS data and metrics from telemetry events and processes, among other things.

Here is an example of the dashboard in action:

AppSignal

AppSignal is another great tool for collecting performance data (among other things). Adding AppSignal to an existing application takes a few seconds. Just install the appsignal dependency and run the appsignal.install mix task with an API key. It has a good set of default metric collection including the throughput and response times for the application. You could even set up minutely probes to track custom metrics. Here's an example of the AppSignal dashboard collecting data from a sample application:

If you are looking to collect data about specific blocks of code that you suspect are slow, AppSignal's instrumentation feature can help collect this data. Just wrap the suspected code inside Appsignal.instrument/2, like this:

defmodule MyApp.PageController do
  use MyApp, :controller

  def index(conn, _params) do
    custom_function()
    render(conn, "index.html")
  end

  defp custom_function do
    Appsignal.instrument("custom_function", fn ->
      :timer.sleep(1000)
    end)
  end
end

This will show up on the AppSignal event graph tagged as custom_function. You can then explore this event across multiple calls and make an informed decision about the need for optimization.

Check out the full list of features in the AppSignal docs.

Wrap-up: Measure Your Elixir App's Performance with Profiling

In this post, we saw how inbuilt tools like cprof, eprof, and fprof can help gather performance insights for your code, even during production.

We also took a quick look at a couple of other tools to monitor and trace your applications: Phoenix Live Dashboard and AppSignal.

Until next time, happy profiling!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Phoenix LiveView Under The Hood: The Form Function Component

Roy Tomeij — 2022-03-29T00:00:00+00:00

Thanks to HEEx and function components, LiveView provides developers with a sleek, ergonomic syntax for building and maintaining sophisticated interactive UIs. LiveView's form/1 function component is a great example of this, making it easier than ever before to render complex forms within LiveView.

However, the form/1 function component can feel a little mysterious to anyone unfamiliar with LiveView's function components.

In this post, we'll look under the hood of the form/1 function component. We'll dive into the Phoenix Component functionality that underpins this function and explore features like component slots. When we're done, you'll know exactly how the form/1 function renders your forms and you'll have a deeper understanding of LiveView components, setting you up to build your own components in the future.

What Are Function Components in Phoenix LiveView?

Before we dive into the form/1 function component, let's discuss LiveView's function components at a high level. Function components are defined in modules that use the Phoenix.Component behavior. Any function that takes in an argument of assigns and returns some markup wrapped in a HEEx template is a function component.

Let's take a look at a simple example. We'll define a UserDetails module that implements a function component, contact_info.

defmodule MyAppLive.UserDetails do
  use Phoenix.Component

  def contact_info(assigns) do
    ~H"""
    <div class="contact-info">
      <p><strong>Phone:</strong><%= @user.phone_number %></strong></p>
      <p><strong>Email:</strong><%= @user.email %></strong></p>
    </div>
    """
  end
end

Now, we can call on this component with any HEEx template like this (assuming you have an available @current_user assigns):

<UserDetails.contact_info user="{@current_user}" />

If you import the UserDetails module or if you're calling on this function component somewhere where the function is defined locally (i.e., elsewhere in the UserDetails module), you can leave off the module name from the function call:

<.contact_info user={@current_user}>

With this basic understanding of function components under your belt, we're ready to dive into the .form/1 function component that LiveView makes available to you. We'll take a look at some more advanced features of function components along the way.

Calling the `form/1` Function Component

LiveView implements a function component, form/1. This function component is defined in Phoenix.LiveView.Helper and imported into all of your live views. We'll start with the entry point of the form/1 code flow—the caller.

In this example, we'll construct a form for a book record that has a title, description, and author, like this:

<.form
    let={f}
    for={@changeset}
    id="book-form"
    phx-change="validate"
    phx-submit="save">

  <%= label f, :title %>
  <%= text_input f, :title %>
  <%= error_tag f, :title %>

  <%= label f, :description %>
  <%= textarea f, :description %>
  <%= error_tag f, :description %>

  <%= label f, :author %>
  <%= select f, :author_id, Enum.map(@authors, &{&1.full_name, &1.id}) %>
  <%= error_tag f, :author_id %>

  <div>
    <%= submit "Save", phx_disable_with: "Saving..." %>
  </div>
</.form>

Already, we can see that the form/1 function component offers a clean and easy-to-read syntax for describing forms. Let's break down the function invocation here. Then, we'll trace what's happening under the hood.

The form/1 function component, like all function components, is called with one argument of the assigns. Some of the attributes passed as part of the assigns probably look familiar from working with Phoenix.HTML.form_for/4. We pass in the changeset, an ID, and some LiveView form bindings. In fact, we can optionally pass in any of the same options you would give to Phoenix.HTML.form_for/4, and the form/3 function component will use those options in the same way.

In addition to the form options we're giving to assigns, we also use the let assigns to tell the function component to give a value back to the caller. More on this in a bit.

Lastly, you'll notice that we're actually calling the form/1 function component with an opening and closing <.form> tag, and we've enclosed the content of our form within those tags. This eloquent, declarative syntax is made possible by the Phoenix Component's render_slot/2 functionality. We'll see that in action later on in this post.

Now that we've seen how to call on the form/1 function component, let's dive into the function's implementation.

The `form/1` Component Under the Hood

You already know that a function component takes in an argument of some assigns and returns some markup wrapped in a HEEx template. So, you won't be surprised to see that at its core, the form/1 function component sets up a Phoenix.HTML.Form struct and renders it in a HEEx template that returns an HTML form.

We'll break down this process one step at a time, beginning with the creation of the form struct.

Creating the `Phoenix.Form`

The first thing that the form/1 function does under the hood is to construct a Phoenix.HTML.Form struct. This is constructed using the data in the assigns that we passed into our function call. Let's take a look at the code:

# Extract options and then to the same call as form_for
action = assigns[:action] || "#"
form_for = assigns[:for] || raise ArgumentError, "missing :for assign to form"
form_options = assigns_to_attributes(assigns, [:action, :for])

# Since FormData may add options, read the actual options from form
%{options: opts} =
  form = %Phoenix.HTML.Form{
    Phoenix.HTML.FormData.to_form(form_for, form_options)
    | action: action
  }

First, it casts the data from the assigns into the same format that Phoenix.HTML.form_for/4 uses to construct form structs. Then, it initializes a new struct.

Next up, the function constructs the form method, CSRF token, and multi-part setting from the data in the form struct, like this:

# And then process method, csrf_token, and multipart as in form_tag
{method, opts} = Keyword.pop(opts, :method, "post")
{method, hidden_method} = form_method(method)

{csrf_token, opts} =
  Keyword.pop_lazy(opts, :csrf_token, fn ->
    if method == "post", do: Plug.CSRFProtection.get_csrf_token_for(action)
  end)

opts =
  case Keyword.pop(opts, :multipart, false) do
    {false, opts} -> opts
    {true, opts} -> Keyword.put(opts, :enctype, "multipart/form-data")
  end

With the form struct and options in place, we then move on to constructing the assigns that will be rendered in the HEEx template here:

assigns =
  LiveView.assign(assigns,
    form: form,
    csrf_token: csrf_token,
    hidden_method: hidden_method,
    attrs: [action: action, method: method] ++ opts
  )

For LiveView to track changes to assigns values rendered by a function component, it must render a valid assigns either passed in as the only argument given to the function or created via a call to Phoenix.LiveView.assign/3 or Phoenix.LiveView.assign_new/3. So, the function component uses LiveView.assign/3 here to construct a new set of assigns to render in the HEEx template.

With the assigns in place, the function will then render the template. Let's take a look at that now.

Rendering the HEEx Template

The form/1 function component, like all function components, returns some markup wrapped in a HEEx template. Let's take a look at that return value now:

 ~H"""
<form {@attrs}>
  <%= if @hidden_method && @hidden_method not in ~w(get post) do %>
    <input name="_method" type="hidden" value={@hidden_method}>
  <% end %>
  <%= if @csrf_token do %>
    <input name="_csrf_token" type="hidden" value={@csrf_token}>
  <% end %>
  <%= render_slot(@inner_block, @form) %>
</form>
"""

Unsurprisingly, the form/1 function component simply returns a HEEx template wrapping a call to an HTML <form>. Let's dig into how the template uses some of the assigns established in the previous step.

First up, you can see that the @attrs assigns is interpolated directly into the opening <form> tag. This will render the tag with the appropriate method= and action= attributes. Next up, you can see that the @hidden_method assigns determines if and how to populate the hidden input. Then the @csrf token, if present, is added to the HTML form in another hidden input.

Most of this template has been pretty straightforward so far. The HEEx template renders, and HTML is constructed from various assigns values. Next up, we'll look at how the form fields we specified between our opening and closing <.form> tags are rendered into the template with the component slot functionality.

Render the Inner Block with Phoenix Component Slots

Phoenix Components implement a feature called "slots". Slots enable us to give blocks to our function component calls, nesting them within opening and closing function component tags like regular HTML tags. Slots are the reason for the form/1 syntax we used above:

<.form ...assigns>
  <!-- form fields -->
</.form>

Here's how it works. Suppose you call on a function component with opening and closing function component tags. That function component's template renders the content in those tags with the help of the render_slot/2 function.

Let's take a look at an example. Recall the UserDetails.contact_info/1 function component we defined earlier:

defmodule MyAppLive.UserDetails do
  use Phoenix.Component

  def contact_info(assigns) do
    ~H"""
    <div>
      <p><strong>Phone:</strong><%= @user.phone_number %></p>
      <p><strong>Email:</strong><%= @user.email %></p>
    </div>
    """
  end
end

We can refactor it to use slots like this:

defmodule MyAppLive.UserDetails do
  use Phoenix.Component

  def contact_info(assigns) do
    ~H"""
    <div class="contact-info">
      <%= render_slot(@inner_block) %>
    </div>
    """
  end
end

Now, we have a dynamic function component we can use to render any type of contact info:

<UserDetails.contact_info user="{@current_user}">
  <p><strong>Phone:</strong><%= @current_user.phone_number %></p>
</UserDetails.contact_info>

Or:

<UserDetails.contact_info user={@current_user}>
  <p>
    <strong>Address:</strong>
    <ul>
      <li><%= @current_user.street_address %></li>
      <li><%= @current_user.city %></li>
      <li><%= @current_user.zip_code %></li>
      <li><%= @current_user.country %></li>
    </ul>
  </p>
</UserDetails.contact_info>

The HTML markup we include between the opening and closing function component tags becomes available in the function component as the @inner_block assigns. The render_slot/2 function renders that content for us.

We can clean this up even further with the let assigns to yield a variable from the function component's template back to the calling template. Let's take a look.

First, we'll call the function component with an assigns of let set equal to a variable, user:

<UserDetails.contact_info let="{user}" user="{@current_user}">
  <!-- coming soon -->
</UserDetails.contact_info>

Then, we'll update the function component's call to render_slot/2 by invoking it with a second argument of the @user assignment, like this:

def contact_info(assigns) do
  ~H"""
  <div class="contact-info">
    <%= render_slot(@inner_block, @user) %>
  </div>
  """
end

The function component sets the variable we specified in the let assignment equal to the value of whatever is passed in as the second argument to render_slot/2. Now, the inner content between the function component's opening and closing tags in the calling template has access to the user variable. We can update our inner block to look like this:

<UserDetails.contact_info let="{user}" user="{@current_user}">
  <p><strong>Phone:</strong><%= user.phone_number %></p>
</UserDetails.contact_info>

With this basic understanding of slots in place, let's revisit the form/1 template:

~H"""
<form {@attrs}>
  <%= if @hidden_method && @hidden_method not in ~w(get post) do %>
  <input name="_method" type="hidden" value="{@hidden_method}" />
  <% end %> <%= if @csrf_token do %>
  <input name="_csrf_token" type="hidden" value="{@csrf_token}" />
  <% end %> <%= render_slot(@inner_block, @form) %>
</form>
"""

Here, its calling render_slot/2 with the @inner_block assignment and the @form assignment. Recall that we invoked form/1 like this:

<.form
    let={f}
    for={@changeset}
    id="book-form"
    phx-change="validate"
    phx-submit="save">

  <%= label f, :title %>
  <%= text_input f, :title %>
  <%= error_tag f, :title %>

  <!-- ... -->
</.form>

So, the form fields render as the @inner_block assignment. The call to <.form> establishes a variable f, which gets set to a value of the second argument passed to render_slot/2. The second argument to render_slot/2 is the @form assignment, pointing to our Phoenix.HTML.Form struct. So, the inner content in our calling template can use the f variable to reference the form struct and build out the Phoenix form to render.

And that's it!

Wrap Up: Demystifying Phoenix LiveView's Form Function Component

While the call to the form/1 function component can seem mysterious, tracing the code under the hood isn't too daunting.

To recap: we can see that the function establishes a Phoenix form and assembles some assigns. Then, it returns a HEEx template that renders an HTML form with the assigns. The HEEx template uses Phoenix Component's slot functionality to render the inner content of our specified form fields, and yields the Phoenix form back to the calling template where we construct those fields.

I hope that this dive under the hood of the form/1 function component has not only demystified that function, but also given you a deeper understanding of how you can use function components and slots in your own live views. Now, you're ready to build out your own extensible function component that dynamically renders different inner blocks of content.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Guide to Secure Elixir Package Updates

Roy Tomeij — 2022-03-15T00:00:00+00:00

Keeping your dependencies up-to-date is essential to ensure that your applications stay healthy, secure, and performant. Thankfully, the BEAM ecosystem has its own package manager, Hex, which is fast, mature, and simple to use.

This article explores the available tools and commands to manage Hex dependencies and some tips to make the process more enjoyable.

Let's dive in!

List Updatable Dependencies in Your Elixir App

You can use the commands below to understand the relationships between dependencies before you attempt to update any of them.

List all your application's dependencies with:

mix deps --all

* bunt 0.2.0 (Hex package) (mix)
  locked at 0.2.0 (bunt) 7af5c7e0
  ok
* castore 0.1.15 (Hex package) (mix)
  locked at 0.1.15 (castore) c69379b9
  ok
* connection 1.1.0 (Hex package) (mix)
  locked at 1.1.0 (connection) 722c1eb0
  ok
* cowboy 2.9.0 (Hex package) (rebar3)
  locked at 2.9.0 (cowboy) 2c729f93
  ok
* cowboy_telemetry 0.4.0 (Hex package) (rebar3)
  locked at 0.4.0 (cowboy_telemetry) 7d98bac1
  ok

Or you can choose to print them in a tree format with:

mix deps.tree

This produces output like the following:

short
├── credo ~> 1.6 (Hex package)
│   ├── bunt ~> 0.2.0 (Hex package)
│   ├── file_system ~> 0.2.8 (Hex package)
│   └── jason ~> 1.0 (Hex package)
├── ecto_psql_extras ~> 0.6 (Hex package)
│   ├── ecto_sql ~> 3.4 (Hex package)
│   ├── postgrex >= 0.15.7 (Hex package)
│   └── table_rex ~> 3.1.1 (Hex package)

To produce an image output, run:

mix deps.tree --format dot && dot -Tpng deps_tree.dot -o deps_tree.png

Note: This option requires Graphviz. Read these Graphviz instructions to install it on your system.

Then open the created deps_tree.png file with a viewer of your choice. Being able to quickly visualize dependencies can help you decide whether a package is worth keeping in your mix.lock. A package could be pulling too many sub-dependencies or might not even be used in your app. Remember, the fewer dependencies, the easier it is to keep things up-to-date.

When you remove a dependency from mix.exs, it will remain in mix.lock. To remove unused dependencies, run:

mix deps.clean --unlock --unused

Check for Outdated Dependencies with Hex

After making sense of your dependencies tree and performing any necessary cleanups, check for outdated packages with:

mix hex.outdated --all

The output will resemble:

Dependency              Current  Latest  Status
bunt                    0.2.0    0.2.0   Up-to-date
cowlib                  2.11.0   2.11.0  Up-to-date
credo                   1.6.1    1.6.3   Update possible
db_connection           2.4.1    2.4.1   Up-to-date
decimal                 2.0.0    2.0.0   Up-to-date
earmark_parser          1.4.19   1.4.20  Update possible
postgrex                0.15.13  0.16.2  Update not possible

To view the diffs in each available update, visit:
https://hex.pm/l/AsY7q

Note: The --all flag shows all outdated packages, including the children of packages defined in mix.exs.

Notice the link in the output above. Hex prepares a nice page for us to inspect the diffs:

Pro-Tip

Use the --within-requirements flag in your CI to notify you of available updates.

mix hex.outdated --within-requirements 1>/dev/null || echo 'Updates available!'

The output of Elixir's package management tasks tends to be concise, well-documented and precisely guides you towards actions.

Inspecting Changes with Hex

To see changes between two package versions in the terminal, run:

# This will display the diff for the package opus
mix hex.package diff opus 0.7.0 0.8.1

diff --git a/var/folders/nc/t881hgn/T/opus-0.7.0-9F0FC44B/mix.exs b/var/folders/nc/t881/T/opus-0.8.1-2BEF6AED/mix.exs
index 0aba420..0c1ad3e 100644
--- a/var/folders/nc/t881/T/opus-0.7.0-9F0FC44B/mix.exs
+++ b/var/folders/nc/t881/T/opus-0.8.1-2BEF6AED/mix.exs
@@ -4,7 +4,7 @@ defmodule Opus.Mixfile do
   def project do
     [
       app: :opus,
-      version: "0.7.0",
+      version: "0.8.1",
       elixir: "~> 1.6",
       elixirc_paths: elixirc_paths(Mix.env()),
       build_embedded: Mix.env() == :prod,
@@ -39,7 +39,7 @@ defmodule Opus.Mixfile do
   defp deps do
     [
       {:retry, "~> 0.8"},
-      {:telemetry, "~> 0.4", optional: true},
+      {:telemetry, "~> 0.4 or ~> 1.0", optional: true},
       {:credo, "~> 0.8.10", only: [:dev, :test], runtime: false},
       {:ex_doc, "~> 0.24.2", only: :dev, runtime: false},
       {:dialyxir, "~> 1.0.0-rc.3", only: [:dev, :test], runtime: false},

You can view the diff in the browser by navigating to:

https://diff.hex.pm/diff/<package_name>/<version1>..<version2>

For example: https://diff.hex.pm/diff/opus/0.7.0..0.8.1

Hex Diff generates a highlighted git diff which you can view in the browser. You can share the link or even highlight a specific row.

Third-party dependencies are essentially somebody's code downloaded from the internet, which ends up in your application. There is no shortage of examples where packages have been hijacked and malicious versions uploaded.

Ideally, you should inspect the diff of every update. Hex seems to be the only package manager with this built-in feature at the moment.

Browsing Changelogs

Ultimately, an update might be available, but is it safe to apply it? Are there any code or configuration changes required for the update to work without issues? The diff between two package versions may contain thousands of lines of templates, tests, and docs that might not seem relevant to you.

Furthermore, a package might not even follow Semver (semver indicates whether the update is safe in compatibility terms).

Commonly, package maintainers keep a changelog to communicate notable changes and upgrade paths concisely. Read more about the benefits of keeping a changelog.

Now the bad news: not all packages have a changelog. So let's go changelog hunting!

The following task will fetch information for the credo package:

mix hex.info credo

A static code analysis tool with a focus on code consistency and teaching.

Config: {:credo, "~> 1.6"}
Locked version: 1.6.3
Releases: 1.6.3, 1.6.2, 1.6.1, 1.6.0, 1.6.0-rc.1, 1.6.0-rc.0, 1.5.6, 1.5.5, ...

Licenses: MIT
Links:
  Changelog: https://github.com/rrrene/credo/blob/master/CHANGELOG.md <--- Here
  GitHub: https://github.com/rrrene/credo

As you can see, the maintainer has added a link to the changelog, so that's nice of them.

There is even a link to the changelog in the HexDocs, which some developers may find really handy:

Tips:

Ensure there's a link to your changelog in mix.exs
Include the changelog in the hexdocs

Automated Changelog Fetching

Hunting for changelogs can get tedious after a while, especially if you want to update many packages. Thankfully, there is now an experimental package for that.

You can add it in your dependencies with:

defp deps do
  [
    {:changelog, "~> 0.1", only: [:dev, :test], runtime: false}
  ]
end

Ensure it is fetched:

mix deps.get

Invoke it for all updatable packages:

mix changelog

Or for a number of packages:

mix changelog tailwind jason

Package: tailwind
Current version: 0.1.4
Latest version:  0.1.5
Hexdiff: https://diff.hex.pm/diff/tailwind/0.1.4..0.1.5

## v0.1.5 (2022-01-18)
  * Prune app.js css import to remove required manual step on first install

Note: This will only print the version to update to, a link to the diff, and the changelog when there is a more recent version.

I wrote this open-source task. It uses some heuristics to locate the changelog by retrieving the Hex package metadata from the API, falling back to common locations in the repo.

My wish is that Hex will standardize including a link to a changelog in mix.exs and mix hex.info, and that Hex Diff will be enhanced to include changelog information.

Updating Dependencies in Elixir

To update all dependencies, run:

mix deps.update --all

In the output, you will see version updates in the following format:

Upgraded:
  credo 1.6.1 => 1.6.3
  earmark_parser 1.4.19 => 1.4.20
  ecto_sql 3.7.1 => 3.7.2
  ex_doc 0.27.3 => 0.28.2 (minor)
  makeup 1.0.5 => 1.1.0
  mint 1.4.0 => 1.4.1
  nimble_parsec 1.2.0 => 1.2.2
  nimble_pool 0.2.5 => 0.2.6
  phoenix_live_dashboard 0.6.2 => 0.6.5
  phoenix_live_view 0.17.5 => 0.17.7
  phoenix_view 1.1.0 => 1.1.2
  plug 1.12.1 => 1.13.3
  postgrex 0.15.13 => 0.16.2 (minor)
  tailwind 0.1.4 => 0.1.5
New:
  hpax 0.1.1

This is the same as running:

mix deps.unlock --all && mix deps.get

Keep in mind that this task will try to upgrade to versions that match the specifications in your mix.exs.

For example:

def deps do
  [
    {:some_package, "~> 0.9"}
  ]
end

Even if there is a more recent 1.0 version for some_package, it does not match the specification above, and it won't be upgraded. You will have to change your mix.exs and try again.

As we saw in the first section of this article, there is a task you can run to check if an update is possible for a package.

mix hex.outdated some_package

Dependency              Current  Latest  Status
some_package            0.9.0    1.0.0   Update not possible

There might be a conflict in some cases where the package resolution cannot find a version to satisfy the dependencies in mix.exs.

A workaround (that you should use with caution) is override.

def deps do
  [
    {:some_package, "~> 1.0"},
    {:other_package, "~> 2.0", override: true}
  ]
end

In this case, the dependency will override any other definitions by other dependencies.

Functional Programming in Elixir with Witchcraft

Roy Tomeij — 2022-02-08T00:00:00+00:00

While Elixir is a functional programming language, it is different from most of the other popular functional languages like Haskell, Scala, OCaml, and F#.

Elixir pragmatically handles concurrent systems with high fault tolerance. In other words, Elixir is an FP language because this naturally fits it, and not for its own sake. So, porting idioms blindly from Haskell to Elixir can lead to undesired results.

At the same time, Elixir users should more frequently visit the whole wonderful universe of functors, monads, and other curiosities. Good libraries have been introduced for operating with algebraic structures.

In this article, I want to introduce you to a library called Witchcraft and show how you can use it to emulate Haskell-style programming in Elixir.

What is Witchcraft?

Witchcraft is a library written by Brooklyn Zelenka that provides Elixir programmers with predefined abstractions for writing Haskell-like code. Witchcraft takes into account the eccentricities of Elixir, such as the frequent use of the pipe operator.

In other words, Witchcraft is a library for writing Haskell 'fan fiction' in Elixir.

It connects with a few other libraries. All in all, the whole Witchcraft complex consists of four parts:

Quark provides some functional building blocks, including flipping arguments of a function in place, a macro for currying functions, and other commonly used things.
TypeClass provides a good interface for generating type classes and their instances.
Witchcraft implements common algebraic and categorical abstractions, such as monads, functors, etc.
Algae implements a domain-specific language (DSL) for creating composite data types like you would in Haskell, and also contains commonly-used composite data types.

Now, let’s try working with the library.

String Validation and Processing with Witchcraft

We will go through a credit card validation exercise.

Given a string that should contain a credit card number, we need to make sure that:

it contains 16 digits
it has at least two different digits
the final digit is even
the sum of the digits is more than 16

We will do this exercise with Either, one of the predefined data types that the Witchcraft complex offers.

Set Up Witchcraft

First, add the necessary libraries to your mix.exs file and run mix deps.get.

  defp deps do
    [
      {:witchcraft, "~> 1.0"},
      {:algae, "~> 1.2"},
    ]
  end

Parsing the String

First, make a new module and import Witchcraft at the top of the module.

defmodule Card do
  use Witchcraft

end

Then, write a function to parse the string into a list of digits and remove all non-digit items.

def get_digits(cardnumber) do
  cardnumber
  |> String.split("", trim: true)
  |> Enum.map(fn x -> Integer.parse(x) end)
  |> Enum.filter(fn x -> x != :error end)
  |> Enum.map(fn {int, _rest} -> int end
end

In the code above, we split the card number into characters and map Integer.parse (safe integer conversion) over them. Integer.parse returns either {int, rest_of_binary} or :error. We filter out the errors and unpack the tuples with the last two functions.

iex(1)> Card.get_digits("3456-3233-5689-4445")
[3, 4, 5, 6, 3, 2, 3, 3, 5, 6, 8, 9, 4, 4, 4, 5]

After that, we can write our first validation function with Either.

Quick Intro to the `Either` Data Type

Either is a type that contains one of two different options: Left or Right. Each of these wraps another type. Left stores failures, and Right stores successes.

-- Either data type as defined in Haskell
data Either a b = Left a | Right b

Usually, we return it from computations that can fail when you want to know what kind of error made them fail.

When working with Witchcraft, you can use the predefined Algae.Either to create new instances of the Either data type. We can do that by using one of two structs: %Algae.Either.Left{left: value} or %Algae.Either.Right{right: value}.

Most likely, you've already run into it somewhere. In other languages, it’s sometimes called Result (in Rust, for example). It’s also structurally identical to the result tuple frequently used in Elixir.

{:ok, result} -> {:right, result} -> %Algae.Either.Right{right: result}
{:error, reason} -> {:left, reason} -> %Algae.Either.Left{left: reason}

In fact, we could technically use the result tuple for our exercise. I've constructed it this way on purpose, so you can explore the machinery of Witchcraft while staying on more or less stable ground.

Example of `Either`

Let’s look at an example of Either.

We have parsed a list of digits from the given string, and now we want to know whether there are 16 items in the list. If we took the easy way out, we could write something like this:

def sixteen_digits(cardnumber), do: Enum.count(cardnumber) == 16

Unfortunately, the function above is not very composable: the return value loses the card number, so we won’t be able to pipe the result to do further computations on it.

Therefore, we want the result to be a more complicated structure such as Either.

If there are 16 digits in the list, we want to return an Either.Right with the list.
If there is a different number of digits, we want to return an Either.Left with :not_16_digits.

And here’s the function to do that:

def sixteen_digits(cardnumber) do
  case Enum.count(cardnumber) do
    16 -> %Algae.Either.Right{right: cardnumber}
    _ -> %Algae.Either.Left{left: :not_16_digits}
  end
end

Let’s try it out in iex:

iex(1)> right_number = Card.get_digits("3456-3233---5689-4445")
[3, 4, 5, 6, 3, 2, 3, 3, 5, 6, 8, 9, 4, 4, 4, 5]
iex(2)> Card.sixteen_digits(right_number)
%Algae.Either.Right{right: [3, 4, 5, 6, 3, 2, 3, 3, 5, 6, 8, 9, 4, 4, 4, 5]}
iex(3)> wrong_number = Card.get_digits("444")
[4, 4, 4]
iex(4)> Card.sixteen_digits(wrong_number)
%Algae.Either.Left{left: :not_16_digits}

Other Validation Functions

In the same way, we can make a validation rule for each requirement.

I suggest you try doing it yourself first since they will be similar to the first example.

To remind you, here are the conditions we need to check:

The number has at least two different digits.
The final digit of the number is even.
The sum of the digits is more than 16.

Each of the conditions should have its own function. Each of the functions should take a list of numbers and return:

%Algae.Either.Right{} with the list inside if the condition is fulfilled
%Algae.Either.Left{} with the reason inside if the condition is not fulfilled

If you are ready to proceed, here are the functions that I will use further on in this article:

def two_unique_digits(cardnumber) do
  unique =
    cardnumber
    |> Enum.uniq()
    |> Enum.count()

  case unique >= 2 do
    true -> %Algae.Either.Right{right: cardnumber}
    _ -> %Algae.Either.Left{left: :not_2_unique_digits}
  end
end

def final_even(cardnumber) do
  last_digit = Enum.at(cardnumber, -1)

  case rem(last_digit, 2) do
    0 -> %Algae.Either.Right{right: cardnumber}
    _ -> %Algae.Either.Left{left: :last_not_even}
  end
end

def sum_greater_than_16(cardnumber) do
  case Enum.sum(cardnumber) >= 16 do
    true -> %Algae.Either.Right{right: cardnumber}
    _ -> %Algae.Either.Left{left: :sum_smallr_than_16}
  end
end

Connecting the Functions

But how can we join these functions together? Each of them takes a list but returns a struct. We can’t pipe them into each other because their types don’t match up.

To chain them, we need something that knows how to unwrap Algae.Either and apply other functions to what's inside, and Witchcraft has just the thing we need.

definst Witchcraft.Chain, for: Algae.Either.Left do
  def chain(left, _), do: left
end

definst Witchcraft.Chain, for: Algae.Either.Right do
  def chain(%Right{right: data}, link), do: link.(data)
end

From Algae.Either.

chain, which can also be written in Witchcraft as >>> (or bind), is the famous bind operator from Haskell: >>=. Let’s understand what it does.

As defined for Either, it will take a value — either Left or Right — and a function from a regular value to Either.

In the case of Left, it will ignore the function and pass the value further. In the case of Right, it will apply the function to the value inside Right.

In our code, we can conveniently use >>> so that it looks pipe-y.

def parse_number(cardnumber) do
  cardnumber
  |> get_digits()
  |> sixteen_digits()
    >>> fn x -> two_unique_digits(x) end
    >>> fn x -> final_even(x) end
    >>> fn x -> sum_greater_than_16(x) end
end

Unfortunately, the capture syntax (&two_unique_digits/1) doesn’t work here. (The macro will expand the code to nested captures, which Elixir doesn’t allow you to do via &.)

Combining the Either data type and its defined chain function lets us build up a chain of functions. Even though they take a list and return an Either, the functions can be chained to arrive at a result. The whole chain will return the list in case of success and the first encountered error in the case of failure.

We can try it out in iex.

iex(1)> Card.parse_number("4444-444-222-1")
%Algae.Either.Left{left: :not_16_digits}
iex(2)> Card.parse_number("4444-4444-4444-4444")
%Algae.Either.Left{left: :not_2_unique_digits}
iex(3)> Card.parse_number("4444-4444-4444-4435")
%Algae.Either.Left{left: :last_not_even}
iex(4)> Card.parse_number("1020-0000-0000-0000")
%Algae.Either.Left{left: :sum_smaller_than_16}
iex(5)> Card.parse_number("4545-3232-5423-6788")
%Algae.Either.Right{right: [4, 5, 4, 5, 3, 2, 3, 2, 5, 4, 2, 3, 6, 7, 8, 8]}

From this point onward, you can claim to have written monadic code in Elixir.

Tinkering With the Codebase

Here are some minor changes we can make to the code so it looks nicer. These are not essential to your understanding but can give you more practice with the code example in question.

First off, Witchcraft provides a very handy ~> operator that does the same as piping into Enum.map.

We can use it in our get_digits function.

def get_digits(cardnumber) do
  cardnumber
  |> String.split("", trim: true)
  ~> fn x -> Integer.parse(x) end
  |> Enum.filter(fn x -> x != :error end)
  ~> fn {int, _rest} -> int end
end

After that, empty lists will cause some run-time errors with the Enum.at that we used, and having empty inputs probably isn’t part of the plan, so we can reject empty strings right at the start.

def get_digits(cardnumber) do
  digits =
    cardnumber
    |> String.split("", trim: true)
    ~> fn x -> Integer.parse(x) end
    |> Enum.filter(fn x -> x != :error end)
    ~> fn {int, _rest} -> int end

  case digits do
    [] -> %Algae.Either.Left{left: :empty_input}
    _ -> %Algae.Either.Right{right: digits}
  end
end

And finally, there is a macro that you can use in parse_number to simulate a more Haskell-like way of writing a sequence of actions in a certain context. This style is frequently called the do-notation.

def parse_number(cardnumber) do

  monad %Algae.Either.Right{} do
    digits <- get_digits(cardnumber)
    a <- sixteen_digits(digits)
    b <- two_unique_digits(a)
    c <- final_even(b)
    d <- sum_greater_than_16(c)
    return(d)
  end
end

In the code sample above, we provide the kind of container that we will operate with — %Algae.Either.Right{}. Then we can do actions inside it, and the monad macro will automatically chain our functions. In the end, it will return whatever we put in the return statement (but wrapped in the container).

It's very powerful, but also very confusing at first. If it is hard to understand what’s happening here, I encourage you to try out monad [] first since it is very similar to list comprehensions. In fact, if you think of this macro as generalized list comprehensions, you won't err too much.

`Either` and the Result Tuple in Elixir

Either is very similar to the result tuple in Elixir, and I’ve done that on purpose to keep you on somewhat familiar ground.

In fact, you can implement the same thing without using Witchcraft.

defmodule Card2 do

  def get_digits(cardnumber) do
    digits =
      cardnumber
      |> String.split("", trim: true)
      |> Enum.map(fn x -> Integer.parse(x) end)
      |> Enum.filter(fn x -> x != :error end)
      |> Enum.map(fn {int, _rest} -> int end)
  end

  def sixteen_digits(cardnumber) do
    case Enum.count(cardnumber) do
      16 -> {:ok, cardnumber}
      _ -> {:error, :not_16_digits}
    end
  end

  def two_unique_digits(cardnumber) do
    unique =
      cardnumber
      |> Enum.uniq()
      |> Enum.count()

    case unique >= 2 do
      true -> {:ok, cardnumber}
      _ -> {:error, :not_2_unique_digits}
    end
  end

  def final_even(cardnumber) do
    last_digit = Enum.at(cardnumber, -1)

    case rem(last_digit, 2) do
      0 -> {:ok, cardnumber}
      _ -> {:error, :last_not_even}
    end
  end

  def sum_greater_than_16(cardnumber) do
    case Enum.sum(cardnumber) >= 16 do
      true -> {:ok, cardnumber}
      _ -> {:error, :sum_smaller_than_16}
    end
  end

  def parse_number(cardnumber) do
    digits = get_digits(cardnumber)

    sixteen_digits(digits)
    |> chain(&two_unique_digits/1)
    |> chain(&final_even/1)
    |> chain(&sum_greater_than_16/1)
  end

  defp chain({:ok, result}, f), do: f.(result)
  defp chain({:error, _error} = result, _f), do: result
end

Here, we could also have used a with statement, which handles tagged result tuples in a monadic way.

def parse_number_with(cardnumber) do
    with digits <- get_digits(cardnumber),
         {:ok, a} <- sixteen_digits(digits),
         {:ok, b} <- two_unique_digits(a),
         {:ok, c} <- final_even(b),
         {:ok, d} <- sum_greater_than_16(c)
  do
    {:ok, d}
  else
      err -> err
  end
end

But the benefit of Witchcraft is that you get a specific data type with a specific chain function and a large set of types, type classes, and functions, all of which work well with each other to support a particular programming style.

Why Use Witchcraft to Write Haskell-style Elixir?

There are quite a few benefits to using Witchcraft when writing Elixir:

Powerful abstractions - for example, we have seen the monad macro, which generalizes over the for and with statements in Elixir. While for and with statements are monadic, they are basically made to handle just a few types (lists or tuples) in a certain way. This does cover a large part of your everyday needs, but there are other wonderful things you could create by applying the same pattern differently once you see it. For example, Haskell's wiki lists nine standard monads.
Pre-set patterns of code shared between typed functional programming languages. If you know these patterns, you can basically go to any typed FP language, check the syntax for them, and continue working with essentially no productivity loss. It's the opposite of macros: it works the same way everywhere :)
More elegant code - for example, a monadic way of handling tuples lets us escape from writing many ugly nested case statements. While Elixir's with statement already solves this particular problem, there are plenty of other places to apply these tools.

Unfortunately, Elixir isn’t really built for writing code like this. Therefore, using Witchcraft can also result in:

A certain amount of unnecessary ceremony
Performance penalties
Angry stares from people that read your code

In general, the most significant benefit of trying out this programming style in Elixir is to train your programmer brain. Even though your code might not use Witchcraft or exactly follow these patterns as you write Elixir, it is good to know what exists out there.

Summing Up and Further Witchcraft Resources

In this post, we looked at how you can write Haskell 'fan fiction' in Elixir using Witchcraft with a credit card validation code example.

Hopefully, you've seen that Witchcraft contains a ton of stuff, enough to satisfy your curiosity about this kind of programming and more.

But, while it is a good tool for teaching FP concepts, there aren't a lot of tutorials available. Brooklyn Zelenka did a talk about Witchcraft, but there isn't much else out there. If you would like to learn more about this programming style right now, your best bet is to start exploring a language like Haskell, F#, or OCaml (the latter two also have pipes).

Alternatively, you can explore typed BEAM by trying languages like Gleam or Hamler.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Securing Your Phoenix LiveView Apps

Roy Tomeij — 2022-01-25T00:00:00+00:00

LiveView is a compelling choice for building modern web apps. Built on top of Elixir's OTP tooling, and leveraging WebSockets, it offers super fast real-time, interactive features alongside impressive developer productivity.

In this post, we'll show you how to secure your live view routes with function plugs and group live routes in a secure live session.

Let's dive straight in!

Using Live View to Build a Phoenix Web App

We'll be building some authentication and authorization features into a Phoenix web app built with live view.

The Arcade web app presents regular users with many online games to play. Our app has a survey feature that collects users' demographic data and game ratings. It also has an admin dashboard that should only be accessible to app admins to view survey results.

For the purpose of this post, we'll assume that logged-in users can visit the /games index and /games/:id show routes to select and play a game, along with the /survey route to fill out the user survey.

Additionally, admin users should only be able to visit the /admin-dashboard page. We'll assume that these pages and the live views that back them have already been built. Our focus is on introducing the authentication and authorization code we need to secure these live views.

Let's get started.

Protect Sensitive Routes in Your Phoenix LiveView App

This post assumes you've already built your registration and login flows, along with some function plugs for authenticating the current user and storing their token in the Phoenix session. I recommend using the Phoenix Auth generator to generate this code for free.

This generated code ensures that Phoenix will add a key of :current_user to the conn struct and a "user_token" key to the session when a user logs in.

Now, let's start in the router by putting some live routes behind authentication.

If you run the Phoenix Auth generator, you generate a module, ArcadeWeb.UserAuth, that implements a function plug require_authenticated_user/2, shown here:

def require_authenticated_user(conn, _opts) do
  if conn.assigns[:current_user] do
    conn
  else
    conn
    |> put_flash(:error, "You must log in to access this page.")
    |> maybe_store_return_to()
    |> redirect(to: Routes.user_session_path(conn, :new))
    |> halt()
  end
end

The details of this function aren't too important. Just understand that it takes in a first argument of the conn struct and checks for the presence of a :current_user key. If one is found, it returns the conn. If not, then it redirects to the login path.

When the auth generator creates the UserAuth module, it also imports into your router, like this:

# router.ex
import ArcadeWeb.UserAuth

We can create a new router scope using this function plug. Let's require that a user is logged in for access to the scope routes, like this:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]
  # ...
end

Add the product index, show routes, and the /survey route that any authenticated user can currently visit:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]
  live "/products", ProductLive.Index
  live "/products/:id", ProductLive.Show
  live "/survey", SurveyLive
end

Now, when a user visits /products or any other route in our new scope, Phoenix invokes the require_authenticated_user function plug. Believe it or not, that's all we have to do to restrict our live routes to logged-in users.

We can take a similar approach to authorizing admins to visit the /admin-dashboard. We'll add a new function plug to the UserAuth module, like this:

def require_admin_user(%{current_user: current_user} = conn, _opts) do
  if current_user.admin do
    conn
  else
    conn
    |> put_flash(:error, "You must log in to access this page.")
    |> maybe_store_return_to()
    |> redirect(to: Routes.page_path(conn))
    |> halt()
  end
end

def require_admin_user(conn, _opts) do
  conn
    |> put_flash(:error, "You must log in to access this page.")
    |> maybe_store_return_to()
    |> redirect(to: Routes.page_path(conn))
    |> halt()
end

If the function plug is invoked with a conn struct that does not contain the current user, we will redirect to the root path.

If the function plug is called with a conn that contains a current user, we will check if that user is an admin. If so, return the conn, otherwise, redirect. The details of our check for the admin status, current_user.admin, don't really matter here. Your app may implement admin logic differently. The main takeaway is that we now have a function plug that can authorize certain routes by enforcing that the current user is present and an admin.

Let's now use our new function plug in our router. We'll create a second scope with a pipeline that uses the require_admin_user/1 function plug:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_admin_user]
  live "/admin-dashboard", Admin.DashboardLive
end

Great! If a user points their browser at /admin-dashboard, our function plug will be invoked.

We've ensured that our live view routes are secure with nothing more than normal Phoenix auth plugs. We can implement authentication — requiring the presence of a current user — and authorization — requiring that the current user has specific permissions or roles — just like you would for regular Phoenix routes.

Now, let's look at a new LiveView feature for grouping live routes together and a security challenge that it presents.

Group Live Views in a Live Session

You'll use live sessions to group similar live routes with shared layouts and auth logic. Grouping live routes together in a live session means that we can live redirect to those routes from any other route in the same live session group.

A live redirect is a special kind of redirect that leverages the existing WebSocket connection, minimizing network traffic and keeping your live view speedy.

When you live redirect from one live view to another in the same live session, the current live view process terminates. The new live view is mounted over the current WebSocket connection without reloading the whole page.

This works great for live views that share a layout. The shared layout that frames the live view content will stay in place, and only the portion of the page that renders the current live view within that layout will change.

Let's create our first live session group now for the routes behind regular user authentication:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]

  live_session :user, root_layout: {ArcadeWeb.LayoutView, "authenticated.html"} do
    live "/products", ProductLive.Index
    live "/products/:id", ProductLive.Show
    live "/survey", SurveyLive
  end
end

Here, we establish a live session block to contain all of our routes that need to be authenticated for regular users and tell them to share a common layout found in lib/arcade_web/templates/layout/authenticated.html.heex. You might notice that the live_session macro is called with a first argument, :user. We'll see how that comes into play in just a bit.

Whenever a user live redirects from the /products route to /products/:id, for example, the existing WebSocket connection will not terminate. Instead, we'll kill the current live view process, mount the new live view, and only re-render the necessary portions of the page within the shared layout.

Grouping Live Routes: The Security Problem

This approach presents a security challenge. If we re-use the existing WebSocket connection, we won't be sending a new HTTP request, and we won't go through the plug pipeline defined in our router.

So we must perform authentication and authorization in our router to prevent direct navigation to sensitive routes from the browser. We must also ensure that our live views can perform their own authentication and authorization every time they mount (whether due to a user pointing their browser directly at a live route or a live redirect between live routes in a shared live session).

Luckily for us, LiveView presents an API for performing authorization and authentication when the live view mounts, making it easy for us to apply this logic across all live routes in a shared session. Let's take a look.

Protect Live Views When They Mount

The LiveView framework allows us to hook into a callback function that will run whenever a live view mounts. The on_mount/4 lifecycle hook will fire before the live view mounts, making it the perfect place to isolate re-usable auth logic that can be shared among live views in a live session.

Start by defining a module that implements an on_mount/4 function, like this:

defmodule ArcadeWeb.UserAuthLive do
  import Phoenix.LiveView
  alias Arcade.Accounts

  def on_mount(:user, params, %{"user_token" => user_token} = _session, socket) do
    socket =
      socket
      |> assign(:current_user, Accounts.get_user_by_session_token(user_token))
    if socket.assigns.current_user do
      {:cont, socket}
    else
      {:halt, redirect(socket, to: "/login")}
    end
  end
end

This function will be called with:

a first argument of the atom that we passed to our live_session macro
a second argument of any params that were part of the incoming web request
a third argument of the session containing the "user_token" used to identify the current user
a fourth argument of the socket (remember, if you use the Phoenix Auth generator, Phoenix will add this "user_token" to the session when a user logs in.)

We perform some basic authentication here by looking up the current user and assigning the result to the socket. If the socket then contains a current user value rather than nil, we continue. Otherwise, we halt and redirect. Any on_mount/4 function must conform to this API, returning the :cont tuple or the :halt tuple.

Next up, let's tell our live session to apply this on_mount/4 callback to all of the live routes in its grouping:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]

  live_session :user, on_mount: {UserAuthLive, :user}, root_layout: {ArcadeWeb.LayoutView, "authenticated.html"} do
    live "/products", ProductLive.Index
    live "/products/:id", ProductLive.Show
    live "/survey", SurveyLive
  end
end

Whenever there is a live redirect to "/products" route (or any other route in that live session), the given live view will invoke ArcadeWeb.UserAuthLive.on_mount/4 with a first argument of :user and our authentication logic will execute. Furthermore, any live view within the live session will mount with the :current_user already set in its socket assigns, since we're adding it in the on_mount callback.

Let's set up a similar callback for the admin live session. Add this function to UserAuthLive:

def on_mount(:admin, params, %{"user_token" => user_token} = _session, socket) do
  socket =
    socket
    |> assign(:current_user, Accounts.get_user_by_session_token(user_token))
  if socket.assigns.current_user.admin do
    {:cont, socket}
  else
    {:halt, redirect(socket, to: "/")}
  end
end

Here, when on_mount/4 is called with a first argument of :admin, we will authorize the current user and authenticate them as an admin. Let's add this to a new live session for the admin-protected routes now:

 scope "/", ArcadeWeb do
  pipe_through [:browser, :require_admin_user]
  live_session :admin, on_mount: {UserAuthLive, :admin}, root_layout: {ArcadeWeb.LayoutView, "admin.html"} do
    live "/admin-dashboard", Admin.DashboardLive
    # more admin routes
  end
end

Here, we group admin-protected routes with a shared admin layout. And whenever any of the live views in this session block mount, the UserAuthLive.on_mount/4 function will be called with the :admin atom as a first argument. This ensures that only admin users can access those pages, even when live redirected.

Thanks to Elixir's pattern matching, we can group all of our auth-related on_mount/4 callbacks in a shared module and implement however many live_session blocks we need to organize our live views.

Wrap Up: Easily Group Live Views to Secure Your Phoenix LiveView App

In this post, we explored how LiveView allows you to group live routes in a shared session. Grouping enables live views to easily share a layout and implement shared authentication and authorization logic.

Remember, you must authenticate and authorize both your protected routes in the router and your live views when they mount. Reach for function plug pipelines to achieve the former, and live session and the on_mount/4 callback to accomplish the latter. With this combination of tools, you can bulletproof your live views, making them highly secure and capable of sophisticated authorization logic.

I hope you've found this post useful. Until next time: happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Build Interactive Phoenix LiveView UIs with Components

Roy Tomeij — 2022-01-11T00:00:00+00:00

LiveView empowers developers to build interactive, single-page web apps with ease by providing a framework that eliminates the need for guesswork.

In this post, we'll take a look at how you can layer simple, single-purpose functional components to wrap up shared presentation logic. We'll also use more sophisticated live components to craft easy-to-maintain single-page flows that handle complex user interactions.

Along the way, you'll gain a solid understanding of working with HEEx — Phoenix and LiveView's new templating engine — and you'll see some of LiveView's out-of-the-box function components in action.

Let's dive in!

The Feature: Compose a User Survey UI for a Phoenix LiveView App

Before we dive into writing any actual code, let's talk about the feature we'll build. Imagine that you're responsible for a Phoenix web app, Arcade, that provides in-browser games to users. A user can log in, select a game to play, and even invite friends to play games with them.

In this post, we'll build out a "user survey" feature that asks the user to fill out some demographic info about themselves and then provide a rating for each of several games. We'll focus on that second part of the survey — the game rating forms.

Let's lay out what we'll build in a bit more detail. We'll begin with a parent live view that lives at the /survey route, ArcadeWeb.SurveyLive. This live view will render a child functional component, "ratings index". The ratings index component will iterate over the games and show a game rating if one by the current user exists, or a form for a new rating if not. If users haven't completed a game rating, they will see a list of forms to rate each game 1-5 stars. If they have completed some (or all) ratings, they will see those displayed and forms to submit ratings for the games they have not yet rated.

Here's a look at how it will work:

With our plan in place, we're ready to start writing code.

Define the Parent Live View

We'll build a route first, then mount and render the initial live view.

Define the Survey Route

Our first job is to establish a route. The survey will live at /survey, and it should only work for authenticated users so we can deliver a survey to single, identifiable users. We'll tie the route to the yet-to-be-written SurveyLive live view, with the :index live action, like this:

# router.ex
scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]
    live "/survey", SurveyLive, :index
end

This code assumes you've used the Phoenix Auth generator to add authentication to your Phoenix app. The details of authentication aren't important for our purposes here. Just know that the survey route is a protected route that requires an authenticated user. This means that when a logged-in user points their browser at /survey, the SurveyLive view will mount with a session argument that contains a key of "user_token" pointing to a token we can use to identify the current user. The generator also gives us a function, Accounts.get_user_by_session_token(user_token), that we will use to fetch the user for that token.

With our route established, it's time to define the SurveyLive live view.

Mount the Survey Live View

The mount/3 function builds the initial state for SurveyLive. Let's think a bit about that initial state. We need to use the current user to build our survey's demographic and rating portions since a demographic belongs to a user and a rating belongs to a game and a user. So we want to store that user in the live view's state. This way, we can make it available to the rating form component later. Now, let's implement a mount/3 function that adds the current user to socket assigns, like this:

# lib/arcade_web/live/survey_live.ex
defmodule ArcadeWeb.SurveyLive do
  use ArcadeWeb, :live_view

  def mount(_params, %{"user_token" => user_token}, socket) do
    {:ok,
      socket
      |> assign(
          :current_user,
          Accounts.get_user_by_session_token(user_token))}
  end
end

Okay, we're ready to render a simple version of our survey live view.

Render the Survey Live View

We won't provide a render/1 function, instead we'll use a template — lib/arcade_web/live/survey_live.html.heex. Let's keep it simple for now:

<section class="row">
  <h2>Survey</h2>
</section>

Reload your browser, and you'll see the bare-bones template shown here:

We have the basic framework for our survey UI in place. Now, we're ready to build our first function component.

List Ratings in Phoenix LiveView

Before we build our ratings index function component, let's talk about what function components are and how they work. A function component takes in an assigns argument and returns a HEEx template. Function components are implemented in modules that use the Phoenix.Component behaviour, which also gives us a convenient syntax for rendering function components.

We're almost ready to define our function component. Let's take a step back and discuss HEEx.

Understanding HEEx Templates

A HEEx template is any file ending in the .heex extension that is implicitly rendered by a live view, or any markup rendered by a live view or component that is encapsulated in the ~H""", """ tags. The HEEx templating engine is an extension of EEx. Just like EEx templates, HEEx will process template replacements within your HTML code. Everything between the <%= and %> expressions is a template replacement. HEEx will evaluate the Elixir code within those tags and replace them with the result.

HEEx does more than just templating, though. It also:

provides compile-time HTML validations
gives us a convenient component rendering syntax
optimizes the amount of content sent over the wire, allowing LiveView to render only those portions of the template that need updating when state changes

HEEx is the default templating engine for Phoenix and LiveView. Any generated template files in your Phoenix app will be HEEx templates and end in the .html.heex extension. When using inline render/1 functions in your live views, or function components, you'll return HEEx templates with the ~H sigil.

With that basic understanding in place, we're ready to implement the ratings index function component.

Define the Function Component

We'll build a rating index component responsible for orchestrating the state of all the game ratings in our survey. This component will iterate over the games and render the rating details if a user rating exists, or the rating form if it doesn't. The responsibility for rendering rating details will be handled by a "rating show" function component. A live "rating form" component will handle rendering and managing a rating form. More on live components in a bit.

Meanwhile, SurveyLive will continue to be responsible for managing the overall state and appearance of the survey page. The rating index component will receive the list of game ratings to render from the parent live view. The parent live view is responsible for maintaining and updating that list.

In this way, we keep our code organized and easy to maintain because it adheres to the single responsibility principle — each component has one job to do. By layering these components within the parent SurveyLive view, we compose a series of small, manageable pieces into one interactive feature — the user survey page.

We'll begin by implementing the RatingLive.Index function component. Then, we'll move on to the rating show component, followed by the rating form component.

Create a file, lib/arcade_web/live/rating_live/index.ex, and key in the following component definition:

defmodule ArcadeWeb.RatingLive.Index do
  use Phoenix.Component
  use Phoenix.HTML
  alias ArcadeWeb.RatingLive
end

Our function component uses the Phoenix.Component behaviour which we need to render HEEx templates. Any module that implements only function components will use this behaviour. You'll use a different behaviour when building live or stateful components, which we'll do later on in this post.

We're also using the Phoenix.HTML behaviour here to bring in the Phoenix.HTML.raw/1 function that we'll use to render unicode characters — more on that in a bit. Finally, we're aliasing the name of the component module itself so it's easy to ergonomically invoke other function components defined within the module from within our main function component here.

The entry point of our module will be the games/1 function. We'll call on this function component from the parent live view to render the list of games. The function will take in an assigns argument containing the list of games passed in from the parent SurveyLive view. It will return a HEEx template that iterates over that list and renders another function component to show the game rating details if a rating exists and the rating form live component if not. Define that function now, as shown here:

def games(assigns) do
  ~H"""
  <div class="survey-component-container">
    <.heading games={@games} />
    <.list games={@games} current_user={@current_user}/>
  </div>
  """
end

We're composing our games/1 function out of two additional function components — heading/1 and list/1. We call on those function components with the .function_name assigns... /> syntax. This invokes the function component and passes in whatever assigns we provide as the assigns argument to that function.

It's also worth noting the {} interpolation syntax here, instead of the <%= %> EEx tags you might be used to. This is because HEEx, unlike EEx, isn't just responsible for evaluating and templating Elixir expressions into your HTML. It also parses and validates the HTML itself. So you can't use the traditional EEx tags inside HTML tags in a HEEx template. Instead, use curly braces to interpolate values inside HTML tags and function component calls, and use EEx tags when interpolating values in the body, or inner content, of those tags.

Let's build those additional function components now, starting with heading/1:

def heading(assigns) do
  ~H"""
    <h2>
      Ratings
      <%= if ratings_complete?(@games), do: raw "&#x2713;" %>
    </h2>
  """
end

The heading/1 function is pretty small and single-purpose. It renders an <h2> element that encapsulates some text along with a helper function that checks to see if all of the games have a rating by the current user. If so, we render the unicode to a checkmark and the user can see that all of the ratings forms have been completed.

Before we implement this helper function, let's see how we're going to render the index component with a list of games.

When we render this index component from the SurveyLive template, we'll use the SurveyLive view to query for the list of games with ratings by the current preloaded user.

Then, we'll pass that list of games down into the index component. So we can assume that each game in the @games list has its ratings list populated only with a rating from the current user. With that in mind, we can implement the ratings_complete?/1 function to iterate over the list of games and return true if there is a rating for every game. Add in your function now, like this:

 defp ratings_complete?(games) do
  Enum.all?(games, fn game ->
    length(game.ratings) == 1
  end)
end

Now if a user has completed all of the game ratings, they'll see the "Ratings" header with a nice checkmark next to it:

With the heading/1 function component out of the way, let's turn our attention to list/1. Add in this function now:

def list(assigns) do
  ~H"""
  <%= for {game, index} <- Enum.with_index(@games) do %>
    <%= if rating = List.first(game.ratings) do %>
      <h3>Show rating coming soon!</h3>
    <% else %>
      <h3>Rating form coming soon!</h3>
    <% end %>
  <% end %>
  """
end

Here, we use a for comprehension that maps over all of the games in the system, where each game's ratings list contains the single preloaded rating by the given user if one exists. Inside that comprehension, the template will render the rating details if a rating exists or a form for that rating if not. Nesting components in this manner lets the reader of the code deal with a tiny bit of complexity at a time.

We'll dig into this logic a bit more when we're ready to implement these final two components. With the index component out of the way, we are ready to weave it into our SurveyLive template.

Render the Component

The next bit of code we'll write shows how the presentation of our view can change based on the contents of the socket. The SurveyLive view will use the state of the overall survey to control what is shown to the user. This view holds the list of games, and their ratings by the current user, in state. It will pass this list into the RatingLive.games/1 function component as part of the component assigns. The contents of this list will allow the games/1 function component to determine if it should show rating details or a rating form.

Let's update SurveyLive now to query for the list of games and their ratings from the current user and add them to socket assigns, like this:

defmodule ArcadeWeb.SurveyLive do
  use ArcadeWeb, :live_view
  alias Arcade.Survey

  def mount(_params, %{"user_token" => token}, socket) do
    {:ok,
     socket
     |> assign(:current_user, Accounts.get_user_by_session_token(token))
     |> assign(:games, Catalog.list_games_with_user_rating(user))}
  end
end

Assume that the Catalog.list_games_with_user_rating/1 context function returns the list of all games, with only the rating by the given user preloaded, if any.

Now we're ready to render our rating index function component from the SurveyLive template:

<!-- lib/arcade_web/live/survey_live.html.heex -->
  <RatingLive.Index.games games={@games}
      current_user={@current_user} />

Here, we're once again using the <.function_component assigns... /> syntax to render our function component and the {} interpolation syntax for interpolating within tags in a HEEx template.

Now that we're rendering our RatingLive.Index.games/1 function component with the game list, let's build the stateless function component to show the existing rating for a game.

Show a Rating

We're getting closer to the goal of showing ratings, step by step. Remember, we'll show the existing ratings, and forms for ratings, otherwise.

Let's cover the case for ratings that exist first. We'll define a stateless component to show a rating. Then, we'll render that component from within the HEEx template returned by RatingLive.Index.games/1. Let's get started.

Build the Function Component

Create a file, lib/arcade_web/live/rating_live/show_component.ex, and key this in:

defmodule ArcadeWeb.RatingLive.Show do
  use Phoenix.Component
  use Phoenix.HTML
end

We're defining a module that uses the Phoenix.Component behaviour and the Phoenix.HTML behaviour, since we'll once again need support for the Phoenix.HTML.raw/1 function to render unicode characters.

Okay, let's move on to the entry point of our function component, the stars/1 function. We'll call this function from within the HEEx template returned by RatingLive.Index.games/1 with an assigns that includes the given game's rating by the current user.

The stars/1 function will operate on this rating and use some helper functions to construct a list of filled and unfilled unicode star characters. We'll construct that list using a simple pipeline, and then render it in a HEEx template, like this:

def stars(assigns) do
  stars =
    filled_stars(assigns.rating.stars)
    |> Enum.concat(unfilled_stars(assigns.rating.stars))
    |> Enum.join(" ")

  ~H"""
  <div>
    <h4>
      <%= @game.name %>:<br/>
      <%= raw stars %>
    </h4>
  </div>
  """
end

The filled_stars/1 and unfilled_stars/1 helper functions are interesting. Take a look at them here:

def filled_stars(stars) do
  List.duplicate("&#x2605;", stars)
end

def unfilled_stars(stars) do
  List.duplicate("&#x2606;", 5 - stars)
end

Examining our pipeline in the stars/1 function, we can see that we call on filled_stars/1 to produce a list of filled-in, or "checked", star unicode characters corresponding to the number of stars the game rating has. Then, we pipe that into a call to Enum.concat/2 with a second argument of the output from unfilled_stars/1. This second helper function produces a list of empty, or not checked, star characters for the remaining number of stars.

For example, if the number of stars in the rating is 3, our pipeline of helper functions will create a list of three checked stars and two un-checked stars. Our pipeline concatenates the two lists together and joins them into a string of HTML that we can render in the template.

We have everything we need to display a completed rating, so it's time to roll several components up together.

Render the Component

We're ready to implement the next phase of our plan. The RatingLive.Index.games/1 function component iterates over the list of games in the @games assigns. If a rating is present, we show it. Add in the call to our new Show.stars/1 component now, like this:

def list(assigns) do
  ~H"""
  <%= for {game, index} <- Enum.with_index(@games) do %>
    <%= if rating = List.first(game.ratings) do %>
      <Show.stars rating={rating} game={game} />
    <% else %>
      <h3>Rating form coming soon!</h3>
    <% end %>
  <% end %>
  """
end

It's a straight for comprehension with an if statement. If a rating exists, we render the function component by calling on it with the game and rating assigns. If not, we need to render the form. Let's build that form and render it now.

Submit a Rating

Our rating form will display the form and manage its state, validating and saving the rating. We'll need to pass a game and user for our database relationships, and the game's index in the parent LiveView's socket.assigns.games list. We'll use this index later on to update SurveyLive state efficiently.

Build the Rating Form Component

The component will be stateful since it needs to manage the state of the rating form and respond to user interactions to validate form changes and handle the form submission.

A stateful, or live, component is any module that uses the :live_component behaviour and renders a HEEx template. Such modules can implement the live component lifecycle functions, including mount/3, update/2, and render/1, and respond to user events by implementing a handle_event/3 function. Let's define our live component module now. Create a file, lib/arcade_web/live/rating_live/form.ex, and key this in:

defmodule ArcadeWeb.RatingLive.FormComponent do
  use ArcadeWeb, :live_component
  alias Arcade.Survey
  alias Arcade.Survey.Rating
end

This is simple enough, to begin with. We define our module, use the :live_component behaviour, and add in some aliases that we'll need later.

We'll use LiveView's .form/1 function (more on that in a bit) to construct the rating form. This function requires a changeset, so we'll need to store one in our component's state. Here's where the component lifecycle comes into play. When we render a live component, LiveView starts the component in the parent view's process and calls these callbacks, in order:

mount/1 : The single argument is the socket, and we use this callback to set the initial state. This callback is invoked only once, when the component is first rendered from the parent live view.
update/2 : The two arguments are the assigns argument given to live_component/3 and the socket. By default, it merges the assigns argument into the socket.assigns established in mount/1. We'll use this callback to add additional content to the socket each time live_component/3 is called.
render/1 : The one argument is socket.assigns. It works like a render in any other live view.

Stateful components will always follow this process when first mounted and rendered. Then, when the component updates in response to changes in the parent live view, only the update/2 and render/1 callbacks fire. Since these updates skip the mount/1 callback, the update/2 function is the safest place to establish the component's initial state. Let's create our component's update/2 function now, like this:

def update(assigns, socket) do
  {:ok,
    socket
    |> assign(assigns)
    |> assign_rating()
    |> assign_changeset()}
end

def assign_rating(
    %{assigns: %{current_user: user, game: game}} = socket) do
  assign(socket, :rating, %Rating{user_id: user.id, game_id: game.id})
end

def assign_changeset(%{assigns: %{rating: rating}} = socket) do
  assign(socket, :changeset, Survey.change_rating(rating))
end

These reducer functions will add the necessary keys to our socket.assigns. They'll drop in any assigns our parent sends, add a new Rating struct, and finally establish a changeset for the new rating.

There are no surprises here. One reducer builds a new rating, and the other uses the Survey context to build a changeset for that rating. Now, on to render.

With our socket established, we're ready to render. We'll choose a template to keep our markup code neatly compartmentalized. Create a file, lib/arcade_web/live/rating_live/form.html.heex. Add the game title markup followed by the game rating form shown here:

<div class="survey-component-container">
  <section class="row">
  <h4><%= @game.name %></h4>
  </section>
  <section class="row">
    <.form
      let={f}
      for={@changeset}
      phx-change="validate"
      phx-submit="save"
      phx_target={@myself}
      id={@id}>

      <%= label f, :stars%>
      <%= select f, :stars, Enum.reverse(1..5) %>
      <%= error_tag f, :stars %>

      <%= hidden_input f, :user_id%>
      <%= hidden_input f, :game_id%>

      <%= submit "Save", phx_disable_with: "Saving..." %>
    </.form>
  </section>
</div>

The form template is really just a standard Phoenix form, although the syntax for rendering the form may be new to you. The main function in the template is the form/1 function: a function component made available by LiveView under the hood. The form function component returns a rendered HEEx template containing an HTML form built with the help of Phoenix.HTML.Form.form_for/4.

If you're feeling adventurous, you can check out the source code for the form function component. For now, all you really need to know is that calling form/1 returns an HTML form for the specified changeset, with the specified LiveView bindings. Let's take a closer look at how our form is rendered.

Since the form/1 function is built on top of the form_for/4 function, it presents a similar API. Here, we're generating a form for the @changeset assignment that was put in assigns via the update/2 callback.

Then, we bind two events to the form, a phx-change to send a validate event and a phx-submit to send a save event. We target our form component to receive events by setting phx-target to @myself, and we tack on an id.

Note that we've set a dynamic HTML id of the stateful component id, stored in socket assigns as @id. This is because the game rating form will appear multiple times on the page, once for each game, and we need to ensure that each form gets a unique id. You'll see how we set the id assigns for the component when we render it in a bit.

Our form has a stars field with a label and error tag and a hidden field for each user and game relationship. We tie things up with a submit button.

We'll come back to the events a bit later. For now, let's fold our work into the RatingLive.Index.list/1 function component.

Render the Component

The RatingLive.Index.games/1 function component should render the rating form component if no rating for the given game and user exists. Let's do that now.

def list(assigns) do
  ~H"""
  <%= for {game, index} <- Enum.with_index(@games) do %>
    <%= if rating = List.first(game.ratings) do %>
      <Show.stars rating={rating} game={game} />
    <% else %>
      <.live_component module={RatingLive.Form}
                        id={"rating-form-#{game.id}"}
                        game={game}
                        game_index={index}
                        current_user={@current_user } />
    <% end %>
  <% end %>
  """
end

Here, we call on the component with the live_component/1 function, passing the user and game into the component as assigns, along with the game's index in the @games assignment. We add an :id, a requirement of all stateful components. Since we'll only have one rating per component, our id with an embedded game.id should be unique.

The live_component/1 function is a function component made available to us by the LiveView framework. It takes in an argument of some assigns and returns a HEEx template that renders the given component within the parent live view. When using live_component/1 to render a live component, you must specify an assigns of module, pointing to the name of the live component module to mount and render, and an assigns of id, which LiveView will use to keep track of the component. Also, note the {} interpolation syntax we're using — this syntax is required when interpolating within HTML or HEEx tags.

It's been a while since we've looked at things in the browser — but now, if you point your browser at /survey, you should see something like this:

Handle Component Events

We've bound events to save and validate our form, so we should teach our component how to do both. We need one handle_event/2 function head for each of the save and validate events. Let's start with validate:

def handle_event("validate", %{"rating" => rating_params}, socket) do
  {:noreply, validate_rating(socket, rating_params)}
end

We need to build the reducer next:

def validate_rating(socket, rating_params) do
  changeset =
    socket.assigns.rating
    |> Survey.change_rating(rating_params)
    |> Map.put(:action, :validate)

  assign(socket, :changeset, changeset)
end

Our validate_rating/2 reducer function validates the changeset and returns a new socket with the validated changeset (containing any errors) in socket assigns. This will cause the component to re-render the template with the updated changeset, allowing the error_tag helpers in our form_for form to render any errors.

Next up, we'll implement a handle_event/2 function that matches the save event:

def handle_event("save", %{"rating" => rating_params}, socket) do
  {:noreply, save_rating(socket, rating_params)}
end

And here's the reducer:

def save_rating(
         %{assigns: %{product_index: product_index, product: product}} = socket,
         rating_params
       ) do
  case Survey.create_rating(rating_params) do
    {:ok, rating} ->
      product = %{product | ratings: [rating]}
      send(self(), {:created_rating, product, product_index})
      socket

    {:error, %Ecto.Changeset{} = changeset} ->
      assign(socket, changeset: changeset)
  end
end

Here, we attempt to save the form. On failure, we assign a new changeset. On success, we send a message to the parent live view to do the heavy lifting for us. Then, as all handlers must do, we return the socket.

Update the Rating Index

What should happen when the game rating successfully saves? The RatingLive.Index.games/1 function should no longer render the form for that game. Instead, the survey should display the saved rating. This kind of state change is squarely the responsibility of SurveyLive. Our message will serve to notify the parent live view to change.

Here's the interesting bit. All the parent needs to do is update the socket. The RatingLive.Index.games/1 function already renders the right thing based on the content of the assigns that it receives from the parent, SurveyLive. All we need to do is implement a handler to deal with the "created rating" message.

# lib/arcade_web/live/survey_live.ex
def handle_info({:created_rating, updated_product, product_index}, socket) do
  {:noreply, handle_rating_created(socket, updated_product, product_index)}
end

We use handle_info so that the parent live view process, SurveyLive, can respond to the message sent by the child component. Now, our reducer can take the appropriate action. Notice that the message we match has a message name, an updated game, and its index in the :games list. We can use that information to update the game list without going back to the database. We'll implement the reducer below to do this work:

def handle_rating_created(
         %{assigns: %{products: products}} = socket,
         updated_product,
         product_index
       ) do

  socket
  |> put_flash(:info, "Rating submitted successfully")
  |> assign(
    :products,
    List.replace_at(products, product_index, updated_product)
  )
end

The handle_rating_created/3 reducer adds a flash message and updates the game list with its rating. This causes the template to re-render, passing this updated game list to RatingLive.Index.games/1. That function component, in turn, knows just what to do with a game containing a rating by the given user — it will render that rating's details instead of a rating form.

Notice the lovely layering. In the parent live view layer, all we need to do is manage the list of games and ratings. All of the form handling and rating or demographic details go elsewhere.

The end result of a submitted rating is an updated game list and a flash message. Submit a rating, and see what happens:

You just witnessed the power of components and LiveView.

Wrap Up: You've Built a Complex LiveView UI with Components

This post covered a lot of ground. You implemented a sophisticated UI composed from simple layers, all thanks to LiveView components. You can wrap up simple markup with function components, while live components allow you to maintain component state and respond to events. Meanwhile, HEEx templates and LiveView provide some nice semantics for rendering markup and both types of components.

LiveView is growing fast — it's responding to the community's needs and providing even more ergonomic solutions for developing complex interactive single-page apps. Function components and HEEx are just some of the latest features that make LiveView even more enjoyable to use.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Pitfalls of Metaprogramming in Elixir

Roy Tomeij — 2021-11-16T00:00:00+00:00

Welcome back to this final part of our four-part series on metaprogramming in Elixir.

Previously, we explored the various applications of macros.

In this part, we'll delve into common pitfalls that you might encounter when metaprogramming in Elixir.

Common Perils of Macros

According to the official documentation:

Macros should only be used as a last resort. Remember that explicit is better than implicit. Clear code is better than concise code.

While it may be tempting to use metaprogramming for everything, it may not always be the best option.

The Applications of Macros part of this series outlines the majority of use cases of macros.

However, you should only use macros with great caution.

We'll be looking at these three common pitfalls to avoid when using macros:

Injecting unnecessary functions
Over-injecting behavior
Replacing regular functions

Let's kick off by looking at what happens if you inject unnecessary functions into modules with macros.

Note: These points are inspired by Metaprogramming Elixir.

1. Injecting Unnecessary Functions with Macros

While macros can be used to inject functions into a caller, there are times where this is unnecessary.

Let's look at an example:

defmodule CalculatorTransformer do
  defmacro __using__(_) do
    quote do
      def add(a, b), do: a + b
      def subtract(a, b), do: a - b
      def multiply(a, b), do: a * b
      def divide(a, b), do: a / b
    end
  end
end

defmodule Hospital do
  use CalculatorTransformer

  def calculate_cost(suite, procedure) do
    add(suite * 20, multiply(procedure, 5))
  end
end

We inject various calculator functions into Hospital to eliminate the need for a module identifier.

However, the use of add and multiply now seem like they appear from thin air.

The code loses its semantic meaning and becomes harder to understand for first-time readers.

To preserve the semantic meaning of the code, define CalculatorTransformer as a regular module. This module can be imported into Hospital to eliminate module identifiers:

defmodule CalculatorTransformer do
  def add(a, b), do: a + b
  def subtract(a, b), do: a - b
  def multiply(a, b), do: a * b
  def divide(a, b), do: a / b
end

defmodule Hospital do
  import CalculatorTransformer

  def calculate_cost(suite, procedure) do
    add(suite * 20, multiply(procedure, 5))
  end
end

As well as creating unnecessary functions, macros can also over-inject behavior into a module. Let's explore what this means.

2. Macros Over-injecting Behavior

As Elixir injects macros into the callsite, behavior can be over-injected.

Let's go back to the example of BaseWrapper:

What if we left the parsing logic for post? within the __using__ macro?

defmacro __using__(opts) do
  quote location: :keep, bind_quoted: [opts: opts] do
    # ...

    def post?(url, body) do
      case post(url, body) do
        {:ok, %HTTPoison.Response{status_code: code, body: body}} when code in 200..299 ->
          {:ok, body}

        {:ok, %HTTPoison.Response{body: body}} ->
          IO.inspect(body)
          error = body |> Map.get("error", body |> Map.get("errors", ""))
          {:error, error}

        {:error, %HTTPoison.Error{reason: reason}} ->
          IO.inspect("reason #{reason}")
          {:error, reason}
      end
    end
  end
end

There are two issues with this approach:

By testing post?, you test the inheritor rather than BaseWrapper.

As there are multiple inheritors of BaseWrapper and the entire behavior of post? is injected into the inheritor, we have to test every inheritor individually.

This ensures that any inheritor-specific behavior does not modify the behavior of post?.

Failure to do so can lead to lower test coverage.
Ambiguous error reporting.

Any run-time errors raised by post? will be logged under the inheritor, not BaseWrapper.

Therefore, leaving the entire behavior in post? can create confusion.

The original implementation of BaseWrapper moves the bulk of the parsing behavior into the wrapper instead. This implementation is much neater, semantically more meaningful, and readable.

This minimizes the two issues mentioned above, as:

When you test the core behavior of post?, only BaseWrapper.parse_post is tested — not every single inheritor.
Any errors from parsing will be logged under BaseWrapper.

Note: location: :keep works in a similar fashion.

While we've used wrappers in our example of over-injecting behavior, this can equally apply to regular macros.

A rule of thumb is to minimize the amount of behavior in a macro. Once the necessary information/computations that require a macro have been accessed/performed, you should move the remaining behavior out of the macro.

The final pitfall we'll examine is the use of macros when regular functions suffice.

3. Macros Used in Place of Regular Functions

As powerful as they are, you don't always need macros. In some cases, you can replace a macro's behavior with a regular function.

Let's say that behavior that does not require compile-time information (or a macro to perform computation) is placed in a macro, for example:

defmodule Foo do
  defmacro double(x) do
    quote do
      doubled = unquote(x) * 2
      doubled
    end
  end
end

defmodule Baz do
  require Foo

  def execute do
    Foo.double(3)
  end
end

iex(1)> Baz.execute
6

Here, double could have easily been substituted for a regular function.

Its behavior does not require compile-time information nor a macro for computation. It will be injected into Baz and evaluated when execute is called, just like a regular function.

defmodule Foo do
  def double(x), do: x * 2
end

defmodule Baz do
  def execute, do: Foo.double(3)
end

iex(1)> Baz.execute
6

As you can see, defining double as a macro does not pose any benefits over a regular function.

Metaprogramming in Elixir: Further Reading

We have finally come to the end of this investigation into metaprogramming in Elixir!

Remember: with great power comes great responsibility. Misusing metaprogramming can come back to bite you, so tread lightly.

While this series has aimed to explain metaprogramming and its intricacies concisely, it is by no means the "bible" on this topic.

There are many wonderful resources you can use to learn more about metaprogramming in Elixir! Here are just a few:

Written guides

Books

Metaprogramming Elixir*

Talks

ElixirConf 2017 - Don't Write Macros But Do Learn How They Work by Jesse Anderson

(* - *highly recommended*)

Thanks for reading, and see you next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Authorization and Policy Scopes for Phoenix Apps

Roy Tomeij — 2021-11-02T00:00:00+00:00

Authorization (not to be confused with authentication) is vital to every application but often isn't given much thought before implementation. The IETF Site Security Handbook defines authorization as:

The process of granting privileges to processes and, ultimately, users. This differs from authentication in that authentication is the process used to identify a user. Once identified (reliably), the privileges, rights, property, and permissible actions of the user are determined by authorization.

So, in short, authorization is about defining access policies and scoping.

For example, consider a platform like Github. In very simple terms, it must handle which repositories:

a particular user is allowed to read (this is policy scoping)
the user is allowed to write to (authorization)

If you come from the Rails world, you might be familiar with some gems that provide APIs to handle this, the most popular ones being cancancan and pundit.

In today's post, we'll take a close look at the two critical components of authorization — access policies and scoping. I'll show you how you can roll out your own solution for each in Phoenix and how to leverage the Bodyguard library for quick and easy authorization.

Authorization in Phoenix

There are two parts to authorization that we need to keep in mind:

Access policy — Is this user allowed to perform this operation on this resource?
Policy scope — Which resources is this user allowed to see?

While it is definitely possible to roll out something by hand, it usually makes sense not to reinvent the wheel if well-maintained and tested libraries are available. Canada and Bodyguard are two of the more popular ones that I have seen in the community.

Let's see what implementation might look like for our own solution and also with Bodyguard. We will use a CMS example similar to what the official Phoenix Context guide uses. This CMS allows users to create pages and share them with everyone. Only the author should be able to edit, update, or delete a page once created, but everyone else should see the page.

Implementing Access Policies

Getting back to our CMS example — when the user is on a page, we need an access policy that decides if the user is allowed to perform an action (say, edit) on the page.

Roll Your Own Access Policy

The following is what the official Phoenix guide suggests. This is also what most of us would do, were we rolling out our own authorization solution:

defmodule Hello.CMS.PageController do
  plug :authorize_page when action in [:edit, :update, :delete]

  defp authorize_page(conn, _) do
    page = CMS.get_page!(conn.params["id"])

    if conn.assigns.current_author.id == page.author_id do
      assign(conn, :page, page)
    else
      conn
      |> put_flash(:error, "You can't modify that page")
      |> redirect(to: Routes.cms_page_path(conn, :index))
      |> halt()
    end
  end
end

The implementation is straightforward. We use the :authorize_page plug for edit, update, or delete actions. In that plug, we allow the action only if the page's author is the same as the current user. Otherwise, we redirect to an index page that shows an error.

Use Bodyguard

We can also implement an access policy using Bodyguard.Policy behavior. Depending on the level of access scoping you need, this behavior could be placed on the controller or directly on the underlying context. I usually like to define a separate Policy module to handle this and then delegate the methods from the behavior's target:

defmodule Hello.CMS.Policy do
  @behaviour Bodyguard.Policy

  alias Hello.Accounts.User

  # Super Admins can do anything
  def authorize(_action, %User{role: :super_admin}, _params), do: true

  # Users can list/get/create anything
  def authorize(action, %User{role: :user}, _params) when action in ~w[index show create]a, do: true

  # Users can edit/update/delete own pages
  def authorize(action, %User{id: id, role: :user} = user, %{author_id: ^id})
    when action in ~w[update edit delete]a,
    do: true

   # Default blacklist
  def authorize(_action, _user, _params), do: false
end

Here, we see that we defined authorize(action, user, params) that returns true/false to permit (or not) the action on the resource. You can also get additional control on the error messages by returning {:error, reason} instead of just false.

The Bodyguard.Policy behavior expects the callbacks to return:

:ok or true to permit an action.
:error, {:error, reason}, or false to deny an action.

Then, on the context or the controller, all you need is to delegate the authorize method to our Policy module and then call Bodyguard.permit:

defmodule Hello.CMS.PageController do
  plug :authorize_page

  defp authorize_page(conn, _) do
    page = CMS.show(conn.params["id"])

    case Bodyguard.permit(__MODULE__, action, user, page) do
      :ok ->
        assign(conn, :page, page)
      {:error, _reason} ->
        conn
        |> put_flash(:error, "You can't access that page")
        |> redirect(to: Routes.cms_page_path(conn, :index))
        |> halt()
    end
  end

  defdelegate authorize(action, user, params), to: Hello.CMS.Policy
end

To authorize an action, we can call Bodyguard.permit(Hello.CMS.PageController, action, user). Bodyguard.permit/4 also accepts passing a fourth argument's authorization in the actual resource. This form can be used to authorize actions performed on a single resource (for example, update or delete).

Under the hood, Bodyguard.permit calls authorize on the module we provided as the first argument. The return value is then normalized into :ok or {:error, reason} (regardless of whether we were returning true/:ok or false/:error/{:error, reason} from that method).

While delegating authorize from the controller works well, there might be cases when you need similar access control in multiple places. For example, the same resource could be accessed from your controller and through an Absinthe Resolver exposed through the GraphQL API. For such cases, I suggest delegating authorize/3 on the Phoenix context module:

defmodule Hello.CMS do
  # ...
  defdelegate authorize(action, user, params), to: Hello.CMS.Policy
end

Then, when you need to call Bodyguard.permit from your controller (or the Absinthe Resolver), pass in a first argument of that context module:

defmodule Hello.CMS.PageController do
  defp authorize_page(conn, _) do
    page = CMS.show(conn.params["id"])
    case Bodyguard.permit(CMS, action, user, page) do
      :ok ->
        # OK. Render page
      {:error, reason} ->
        # Error. Show flash and redirect
    end
  end
end

It is also very easy to write tests for the above policy. Here's what the tests might look like:

test "allows users to list pages" do
  user = Hello.Accounts.Fixtures.fixture(:user)
  assert :ok = Bodyguard.permit(Hello.CMS, :index, user)
end

test "allows user to update/delete own pages" do
  user = Hello.Accounts.Fixtures.fixture(:user)
  page = page_fixture(user)
  assert :ok = Bodyguard.permit(Hello.CMS, :update, user, page)
end

test "allows user to create pages" do
  user = Hello.Accounts.Fixtures.fixture(:user)
  assert :ok = Bodyguard.permit(Hello.CMS, :create, user)
end

test "does not allow user to update/delete pages of someone else" do
  user1 = Hello.Accounts.Fixtures.fixture(:user)
  user2 = Hello.Accounts.Fixtures.fixture(:user)

  page = page_fixture(user2)

  assert {:error, :unauthorized} = Bodyguard.permit(Hello.CMS, :update, user1, page)
end

test "allows super_admin to do anything" do
  super_admin = Hello.Accounts.Fixtures.fixture(:user_super_admin)
  user = Hello.Accounts.Fixtures.fixture(:user)
  page = page_fixture(user)
  assert :ok = Bodyguard.permit(Hello.CMS, :index, super_admin)
  assert :ok = Bodyguard.permit(Hello.CMS, :create, super_admin)
  assert :ok = Bodyguard.permit(Hello.CMS, :update, super_admin, page)
end

Implementing Policy Scopes

We've now allowed users to access resources based on some attributes. The next part of the puzzle is to handle the listing of resources. As you might have noticed above, we are allowing users to list everything.

But you might have specific business requirements on what a user can and can't list. For example, in the Github example, users can only see public repositories and the repositories that they have access to (e.g., through their organization or team). Similarly, you could set a restriction that users see all published posts but only their own drafts in a CMS.

Roll Your Own Policy Scoping

This just boils down to using the correct queries based on the user and their access rights. For example, a simple implementation inside a context could look like this:

defmodule Hello.CMS do
  def list_pages(%{id: id} = user) do
    Page
    |> where(author_id: ^id)
    |> or_where(state: :published)
    |> Repo.all()
  end
end

It is definitely possible to do this with simple Ecto queries on the accessor methods. However, it usually makes much more sense to centralize these kinds of domain requirements so that future developers don't forget to include one of the requirements while adding a new feature.

Use Bodyguard

With Bodyguard, we can provide default scoping to query items that a user can access. Implement a scope/3 function inside an Ecto.Schema module from the @behaviour Bodyguard.Schema. The function should filter the query down to only include the resources the user is allowed to access. You can also pass custom params when invoking the scoping to provide further filtering.

Here's how it looks in practice:

defmodule Hello.CMS.Page do
  use Ecto.Schema

  alias Hello.Accounts.User

  def scope(query, user, params) do
    scope_published(query, user, params)
  end

  # Signed in users can access published posts or their own posts (in any state)
  defp scope_published(query, %User{role: :user, id: id}, _params) do
    query
    |> where(author_id: ^id)
    |> or_where(state: :published)
  end

  # Super admins can access anything
  defp scope_published(query, %User{role: :super_admin}, _params), do: query

  # Anonymous users can access only published posts
  defp scope_published(query, _user, _params), do: query |> where(state: :published)
end

Now, this can then be used inside your context when querying. For example:

defmodule Hello.CMS do
  def list_pages(user) do
    Page
    |> Bodyguard.scope(user) # <-- defers to Page.scope/3
    |> Repo.all()
  end
end

If we need to add another listing of the pages later, we can simply use the same Bodyguard.scope(query, user) call and know that everything will be appropriately scoped.

Authorization in Phoenix: Further Reading

In this post, we saw how to implement authorization in your Phoenix apps. We focused on a simple CMS example to explore authorization with — and without — external libraries.

I recommend Dockyard's Authorization Considerations For Phoenix Contexts blog post for a more detailed look at rolling out your authorization solution.

If you are looking for external libraries, check out Bodyguard. I have been using it in production for a while and can vouch for its customizability in authorization. It stays out of the way when we don't want it and also clarifies operations that would otherwise be scattered inside the context and controller methods.

I hope you've found this a useful guide that's inspired you to dive into policy scoping and authorization in Phoenix apps.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Use Macros in Elixir

Roy Tomeij — 2021-10-26T00:00:00+00:00

Welcome back to this third part of our four-part series on metaprogramming in Elixir.

Previously, we established the underlying behavior of macros.

Now we will dive into the various applications of macros in Elixir using open-source Elixir libraries.

Let's go!

Using Macro Wrappers to Extend Elixir Module Behavior

Macros can extend module behavior by adding new functions through wrappers that behave similarly to inheritance in object-oriented programming (OOP).

To extend a module, we use use — this is syntactic sugar that performs the following for us:

defmodule Foo do
  defmacro __using__(opts) do
    quote do
      # Extend another module’s behavior here
    end
  end
end

defmodule Bar do
  use Foo
  # equivalent to calling these two expressions
  # require Foo
  # Foo.__using__([])
end

The __using__ callback macro extends the behavior of Bar. It is injected into the callsite and expanded.

We will be using the following terminology:

Wrapper — a module that extends other modules like Foo.
Inheritor — a module that uses wrappers like Bar, inheriting behavior.

Let’s design a wrapper.

Say we are building a website that relies on several APIs to work. Assuming we use an HTTP client like HTTPoison and each API request has common configurations like request/response data type, we can handle the API requests in three ways:

Set up each API request separately.
Compose API requests by chaining functions that configure each request.
Build a wrapper that automatically configures each API request per request.

While each of these approaches has its merits, we will focus on the last approach.

HTTPoison offers a wrapper for building basic API wrappers through HTTPoison.Base. It handles aspects of an API request like setting up the endpoint and parsing the request/response body.

We will build another wrapper on top of this that will enable inheritors to achieve our business requirements:

Parse all request/response bodies to/from JSON.
Generate an API URL, given a base URL and an endpoint for the base URL.
Configure the request headers to accept JSON and set the content type to JSON.
Parse and handle response bodies.

The wrapper should produce inheritors like:

defmodule DogWrapper do
  use BaseWrapper, base_url: "https://api.dogs.com"

  def upload(name, breed) do
    case post?("upload", %{"name" => name, "breed" => breed}) do
      {:ok, response} -> {:ok, response["link"]}
      {:error, error} -> {:error, error}
    end
  end
end

defmodule CatWrapper do
  use BaseWrapper, base_url: "https://api.cats.com"

  def upload(name, breed, talkative) do
    case post?("upload", %{"name" => name, "breed" => breed, "talkative" => talkative}) do
      {:ok, response} -> {:ok, response["url"]}
      {:error, error} -> {:error, error}
    end
  end
end

defmodule AnimalLovers do
  def upload(:dog, name, breed) do
    case DogWrapper.upload(name, breed) do
      {:ok, link} -> redirect(link)
      {:error, error} -> redirect_error(error)
    end
  end

  def upload(:cat, name, breed, talkative) do
    case CatWrapper.upload(name, breed, talkative) do
      {:ok, link} -> redirect(link)
      {:error, error} -> redirect_error(error)
    end
  end
end

The inheritors do not contain any cumbersome API request logic. Instead, they focus on extracting data from the responses and executing the business logic.

Additionally, the wrapper integrates a base URL into the API requests, so inheritors provide only the endpoint.

Now we know what we want to accomplish, let’s build the wrapper.

First, we define our wrapper, calling it BaseWrapper. This module will define the __using__ callback.

opts is a keyword list that will hold the base_url of the inheritor, and it is bound to the quote.

location option for quote ensures that any errors will directly point to the line in the BaseWrapper.

Any behavior defined in quote will be injected into the inheritor.

defmodule BaseWrapper do
  defmacro __using__(opts) do
    quote location: :keep, bind_quoted: [opts: opts] do
      # behavior goes here
    end
  end
end

Then, we extend our inheritor to use the basic behavior from HTTPoison.Base.

# quote
use HTTPoison.Base

Now, we want to retrieve the base URL from opts. We do not need to unquote opts as it was bound.

# quote
base_url = Keyword.get(opts, :base_url)

With that, we can begin configuring our wrapper using HTTPoison.Base.

HTTPoison.Base defines several callbacks used to process requests and responses.

We will override these callbacks to implement the behavior we want.

# quote
# Modify request headers to include necessary information about the API
def process_request_headers(headers) do
  h = [
    {"Accept", "application/json"},
    {"Content-Type", "application/json"}
  ]

  headers ++ h
end

# Generate the URL for the endpoint using a base URL received from the inheritor
def process_url(endpoint), do: URI.merge(URI.parse(unquote(base_url)), endpoint) |> to_string()

# Automatically encode request body to JSON
def process_request_body(body), do: Jason.encode!(body)

# Automatically decode request body from JSON
def process_response_body(body), do: Jason.decode!(body)

Finally, we want to parse responses from POST requests based on the HTTP status code of the response.

# quote
def post?(endpoint, body) do
  # post is available as we have used HTTPoison.Base which will perform the necessary imports for us
  # post receives only the endpoint of the request as process_url prepends the base URL already
  response = post(endpoint, body)
  # We defer the parsing of the response to a function within the BaseWrapper module.
  # Reasons for doing so will be discussed later
  BaseWrapper.parse_post(response)
end

# BaseWrapper
def parse_post({:ok, %HTTPoison.Response{status_code: code, body: body}})
    when code in 200..299 do
  {:ok, body}
end

def parse_post({:ok, %HTTPoison.Response{body: body}}) do
  IO.inspect(body)
  error = body |> Map.get("error", body |> Map.get("errors", ""))
  {:error, error}
end

def parse_post({:error, %HTTPoison.Error{reason: reason}}) do
  IO.inspect("reason #{reason}")
  {:error, reason}
end

Putting all this together, we have a module that looks like this:

defmodule BaseWrapper do
  defmacro __using__(opts) do
    quote location: :keep, bind_quoted: [opts: opts] do
      use HTTPoison.Base

      base_url = Keyword.get(opts, :base_url)

      def process_request_headers(headers) do
        h = [
          {"Accept", "application/json"},
          {"Content-Type", "application/json"}
        ]

        headers ++ h
      end
      def process_url(endpoint), do: URI.merge(URI.parse(unquote(base_url)), endpoint) |> to_string()
      def process_request_body(body), do: Jason.encode!(body)
      def process_response_body(body), do: Jason.decode!(body)

      # Function injected at compile-time to parse and act on responses accordingly
      def post?(url, body) do
        response = post(url, body)
        BaseWrapper.parse_post(response)
      end
    end
  end

  def parse_post({:ok, %HTTPoison.Response{status_code: code, body: body}})
      when code in 200..299 do
    {:ok, body}
  end

  def parse_post({:ok, %HTTPoison.Response{body: body}}) do
    IO.inspect(body)
    error = body |> Map.get("error", body |> Map.get("errors", ""))
    {:error, error}
  end

  def parse_post({:error, %HTTPoison.Error{reason: reason}}) do
    IO.inspect("reason #{reason}")
    {:error, reason}
  end
end

We've used macro wrappers to extend module behavior. Now, let's focus our attention on performing batch imports using wrappers.

Batch Imports with Macro Wrappers

We can use wrappers to perform batch imports in inheritors.

We can define a group of imports/aliases/requires that will be available during compile-time without performing the imports manually in the inheritor.

This is useful when several inheritors require the same set of imports and is best illustrated in Hound — a browser automation testing library:

# lib/hound/helpers.ex
defmacro __using__([]) do
  quote do
    import Hound
    import Hound.Helpers.Cookie
    import Hound.Helpers.Dialog
    import Hound.Helpers.Element
    import Hound.Helpers.Navigation
    import Hound.Helpers.Orientation
    import Hound.Helpers.Page
    import Hound.Helpers.Screenshot
    import Hound.Helpers.SavePage
    import Hound.Helpers.ScriptExecution
    import Hound.Helpers.Session
    import Hound.Helpers.Window
    import Hound.Helpers.Log
    import Hound.Helpers.Mouse
    import Hound.Matchers
    import unquote(__MODULE__)
  end
end

Hound overrides __using__ to import a host of helper modules available to the inheritor.

The inheritors are always going to be test suites written by other developers. It is convenient and easy to maintain imports through use Hound.Helpers.

Next up, wrappers can also override and dynamically generate functions. Let's see how that works.

Macro Wrappers Define Functions to Override

Like a child class overrides a method in its parent class in OOP, wrappers can define some base behavior for functions and allow inheritors to override this behavior.

This is achieved using defoverridable.

Phoenix uses this to great effect by allowing inheritors of Phoenix.Controller.Pipeline to override functions like init and call to include context-specific behavior.

However, if a custom behavior isn't necessary — i.e. the function is not overwritten — the base behavior will be used instead:

# lib/phoenix/controller/pipeline.ex
defmacro __using__(opts) do
  quote bind_quoted: [opts: opts] do
    # ...

    @doc false
    def init(opts), do: opts

    @doc false
    def call(conn, action) when is_atom(action) do
      conn
      |> merge_private(
        phoenix_controller: __MODULE__,
        phoenix_action: action
      )
      |> phoenix_controller_pipeline(action)
    end

    @doc false
    def action(%Plug.Conn{private: %{phoenix_action: action}} = conn, _options) do
      apply(__MODULE__, action, [conn, conn.params])
    end

    defoverridable init: 1, call: 2, action: 2
  end
end

Macro Wrappers Dynamically Generate Functions

You can also use unquote fragments to dynamically define an arbitrary number of functions. These functions are injected into a module during compile-time. This is especially useful when working with dynamic data or data that comes from an API.

This article will focus on the former, as Metaprogramming Elixir includes a wonderful example of dynamically defining functions for API responses under the code/hub/.

Let's say we are building a script that reads a list of groceries and generates functions for each category. These functions will operate on each unique category and the items within it.

For simplicity, each function will be the category's name and print the contents of the category.

The grocery list will use the following format — <category>:<comma separated list>:

// groceries.txt
dinner:carrots,potatoes,curry,chicken cubes
supplies:paper,pencils
school:folder,binder

We can read this file in an Elixir script:

# groceries.exs
defmodule Groceries do
  @filename "groceries.txt"
  for line <- File.stream!(Path.join([__DIR__, @filename]), [], :line) do
    # line represents each line of the groceries list
  end
end

We've extracted the lines of groceries, now we can parse each line:

# for
[category, rest] = line |> String.split(":") |> Enum.map(&String.trim(&1))
groceries = rest |> String.split(",") |> Enum.map(&String.trim(&1))

Then we can define our dynamic functions.

We use unquote as an argument for def. def is also a macro, so expands the unquote along with anything in its body during compilation. This is known as an unquote fragment.

# for
def unquote(String.to_atom(category))() do
  readable_list = Enum.join(unquote(groceries), " and ")
  IO.puts("Groceries in #{unquote(category)} include #{readable_list}")
end

Putting it all together, we get the following script:

# groceries.exs
defmodule Groceries do
  @filename "groceries.txt"
  for line <- File.stream!(Path.join([__DIR__, @filename]), [], :line) do
    [category, rest] = line |> String.split(":") |> Enum.map(&String.trim(&1))
    groceries = rest |> String.split(",") |> Enum.map(&String.trim(&1))

    def unquote(String.to_atom(category))() do
      readable_list = Enum.join(unquote(groceries), " and ")
      IO.puts("Groceries in #{unquote(category)} include #{readable_list}")
    end
  end
end

iex(1)> Groceries.dinner
Groceries in dinner include carrots and potatoes and curry and chicken cubes
:ok
iex(2)> Groceries.school
Groceries in school include folder and binder
:ok

Now let's turn our attention to implementing domain-specific languages (DSLs) using macros.

Implement Domain-Specific Languages using Macros

Domain-Specific Languages (DSLs) are "computer language(s) specialized to a particular application domain."

DSLs can be built and used in Elixir to solve business requirements.

As the official DSL tutorial points out:

You don’t need macros in order to have a DSL: every data structure and every function you define in your module is part of your Domain-specific language.

However, we will cover a simple implementation of a DSL using macros to demonstrate how that would look.

The simplest example of a DSL is from ExUnit, a built-in testing framework (taken from the ExUnit documentation):

defmodule AssertionTest do
  use ExUnit.Case, async: true

  test "the truth" do
    assert true
  end
end

test is a macro that registers a new test case.

These test cases are collated using accumulating module attributes. Then, when the test suite executes, all of the declared test cases are executed.

You can find a simplified implementation and explanation of this DSL in the official tutorial.

Absinthe — an implementation of the GraphQL specification — has a DSL for defining schemas.

# lib/absinthe/schema/notation.ex
defmacro object(identifier, attrs, do: block) do
  {attrs, block} =
    case Keyword.pop(attrs, :meta) do
      {nil, attrs} ->
        {attrs, block}

      {meta, attrs} ->
        meta_ast =
          quote do
            meta unquote(meta)
          end

        block = [meta_ast, block]
        {attrs, block}
    end

  __CALLER__
  |> recordable!(:object, @placement[:object])
  |> record!(
    Schema.ObjectTypeDefinition,
    identifier,
    attrs |> Keyword.update(:description, nil, &wrap_in_unquote/1),
    block
  )
end

object is a macro that generates the GraphQL schema.

We define it as a macro because we're accessing compile-time information through __CALLER__, and we're attempting to work with the module's metadata. Defining it as a macro loads the schemas during compile-time.

Even the routing functions in Phoenix are a DSL.

Alchemist Camp's 'Creating a DSL for our router' video series explains how to implement a routing system similar to Phoenix's.

It's time to move on to Abstract Syntax Trees (ASTs): how they can be traversed and when to use prewalk vs. postwalk macros.

Traversing Abstract Syntax Trees (ASTs)

We introduced ASTs in part one of this series. You can traverse an existing AST to extract information about — or modify — the structure of the AST to:

Improve performance
Simplify the AST
Perform computations based on the structure of the AST

As the modification of ASTs is a niche process and context-dependent, we will focus on how traversal is performed rather than the specific application of traversal.

Going Back to the Roots of ASTs

Before we can understand the traversal of ASTs, we have to get to grips with the core data structure of ASTs.

Source code is parsed into trees — data structures that house hierarchical tree data. They begin with a root node, followed by a set of sub-trees — children nodes.

Each node includes a reference to its children. Each childless node is known as a leaf.

The figure below illustrates the basic anatomy of a tree:

Order of Traversal in ASTs

Now that we know the underlying data structure of an AST, we can tackle the problem of traversing an AST/tree.

Elixir uses depth-first traversal in either pre-order or post-order.

Depth-first traversal follows a node down to its children recursively until reaching a leaf. The sibling of the node is moved to next, and the recursion repeats.

(Source: Wikimedia Commons)

If we use our example, the nodes are visited in the following order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7.

Pre-order and post-order traversal refer to how depth-first traversal occurs:

Pre-order traversal starts from the root node and traverses depth-first through the AST until the right-most leaf is reached.

In our tree, the nodes would be visited in the following order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7.

Post-order traversal starts from the left-most leaf of each sub-tree and traverses in depth-first fashion until we reach the root node.

In our tree, the nodes would be visited in the following order: 2 -> 4 -> 5 -> 7 -> 6 -> 3 -> 1. The traversal happens upwards towards the root node.

Regardless of the traversal order, the traversal occurs recursively until the "end condition" — i.e., right-most leaf for pre-order and root node for post-order.

In Elixir, functions represent each type of traversal:

Macro.prewalk — performs depth-first, pre-order traversal
Macro.postwalk — performs depth-first, post-order traversal
Macro.traverse — performs depth-first traversal and both pre and post-order traversal on the AST

By traversing the tree in a given order, we can extract information about the AST and use it to modify the AST.

We define functions that will be executed recursively on every node during traversal. These functions are similar to map.

The input is the currently visited node, while the output is a modified/untouched node.

Prewalk vs. Postwalk Macros

You might ask: "What is the benefit of using pre-order traversal over post-order traversal, and vice versa?"

In most scenarios, there is no difference between the two as both will traverse the AST.

However, you'll have a preference for postwalk or prewalk if the operation is order-sensitive — i.e., if you require the traversal to start at the root node or the left-most leaf first.

This may happen when the operation aims to match the first node (in order) against a given condition and only perform the operation on that node. We would rather have the traversal find the node quickly to operate as soon as possible.

In this case, prewalk is preferred over postwalk if the node appears at the beginning of the AST.

Macro.prewalk(ast, fn
  {:match, [], args} -> foo(args)
  otherwise -> otherwise
)

Another consideration is unintended infinite recursion. Take this example:

Macro.prewalk({:foo, [], [:bar]}, fn
  {:foo, [], _} -> {:foo, [], [{:foo, [], [:bar]}]}
  otherwise -> otherwise
end)

In this prewalk, the function matches any node that calls the foo function and replaces it with a recursive call to foo, with foo as the argument.

This will produce an infinite AST that, when converted to Elixir code, will look something like foo(foo(foo(...))).

prewalk is recursively executed on an AST — usually until it reaches the right-most leaf, indicating the AST's end. However, as the AST expands infinitely in the above example, there will never be a right-most leaf, hence infinite recursion.

Using postwalk instead avoids this issue as we simply replace the node once and move on upwards to the root node where the recursion will stop.

Macro.postwalk({:foo, [], [:bar]}, fn
  {:foo, [], _} -> {:foo, [], [{:foo, [], [:bar]}]}
  otherwise -> otherwise
end)

{:foo, [], [{:foo, [], [:bar]}]}

So, while the choice of prewalk and postwalk might not matter when an operation is not order-sensitive, postwalk is preferred over prewalk to avoid infinite recursion.

traverse combines prewalk and postwalk, performing both together. This is useful when we want to traverse the AST in both orders.

Ecto uses prewalk to count the number of interpolations within a given expression.

In this case, there isn't a specific reason to choose prewalk over postwalk.

# lib/ecto/query/builder.ex
def bump_interpolations(expr, params) do
  len = length(params)

  Macro.prewalk(expr, fn
    # The following expression matches a pinned variable which is what Ecto relies on for
    # interpolation
    {:^, meta, [counter]} when is_integer(counter) -> {:^, meta, [len + counter]}
    other -> other
  end)
end

Ecto also uses postwalk to expand dynamic expressions.

# lib/ecto/query/builder/dynamic.ex
defp expand(query, %{fun: fun}, {binding, params, subqueries, count}) do
  {dynamic_expr, dynamic_params, dynamic_subqueries} = fun.(query)

  Macro.postwalk(dynamic_expr, {binding, params, subqueries, count}, fn
    {:^, meta, [ix]}, {binding, params, subqueries, count} ->
      case Enum.fetch!(dynamic_params, ix) do
        {%Ecto.Query.DynamicExpr{binding: new_binding} = dynamic, _} ->
          binding = if length(new_binding) > length(binding), do: new_binding, else: binding
          expand(query, dynamic, {binding, params, subqueries, count})

        param ->
          {{:^, meta, [count]}, {binding, [param | params], subqueries, count + 1}}
      end

    {:subquery, i}, {binding, params, subqueries, count} ->
      subquery = Enum.fetch!(dynamic_subqueries, i)
      ix = length(subqueries)
      {{:subquery, ix}, {binding, [{:subquery, ix} | params], [subquery | subqueries], count + 1}}

    expr, acc ->
      {expr, acc}
  end)
end

Another way you can change module behavior using macros is through compile-time hooks — let's take a quick look at those.

Compile-time Hooks with Macros

Compile-time hooks allow the compilation behavior of a module to be modified. Callbacks accompany these hooks.

The two notable hooks to discuss are @before_compile and @after_compile. They are useful when we want to perform computation right before — or right after — module compilation.

For now, let's look at a basic set of examples for each hook.

With @before_compile, as the callback (defmacro __before_compile__(env)) is called right before compilation, the callback must be declared in a separate module from where the hook references it.

If the callback is declared in the same module, the macro will not compile in time.

defmodule Foo do
  defmacro __before_compile__(env) do
    IO.inspect(env)
    nil
  end
end

defmodule Bar do
  @before_compile Foo
end

An exception to this behavior is when Bar is an inheritor of Foo:

defmodule Foo do
  defmacro __using__(_) do
    quote do
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(env) do
    IO.inspect(env)
    nil
  end
end

defmodule Bar do
  use Foo
end

In this example, as @before_compile is injected into Bar, its callback is defined in Foo (a different module). Since use calls require, it ensures that Foo is compiled before Bar. This means Foo.__before_compile__/1 is always available to Bar.

With @after_compile, there isn't a need to declare the callback (defmacro __after_compile__(env, bytecode)) in another module. This is because the module housing the callback is already compiled, so the callback is available.

defmodule Foo do
  @after_compile __MODULE__

  def __after_compile__(env, _bytecode) do
    IO.inspect(env)
    nil
  end
end

Another hook that is worth mentioning briefly is @on_definition, which invokes its callback whenever a function/macro is defined in the current module.

ExUnit uses @before_compile in a test suite to inject a final function — __ex_unit__ — to execute the test suites after they have been collated.

This function must be injected right before compilation when all the test suites are collated.

You may also wish to store a list of data across macro invocation, such as when ExUnit collates test cases and invokes them all at once.

This can be achieved using module attributes. Let's see that in action.

Module Attributes as Temporary Storage

When we set a module attribute to accumulate, any invocation of the module attribute will add the given value to the list, rather than overriding it:

defmodule Foo do
  Module.register_attribute(__MODULE__, :names, accumulate: true)
  @names "John"
  @names "Peter"

  def print, do: IO.inspect(@names)
end

iex(1)> Foo.print
["Peter", "John"]

We may also want to accumulate values through a function call.

This happens when the parent module first compiles, and then a secondary module updates the module attribute via a macro, like so:

defmodule Foo do
  defmacro add(name) do
    quote do
      @names unquote(name)
      :ok
    end
  end
end

defmodule Bar do
  require Foo
  import Foo

  Module.register_attribute(__MODULE__, :names, accumulate: true)

  add "john"
  add "henry"

  IO.inspect(@names) # This prints ["henry", "john"] right after the module is done compiling
end

Foo must compile first. The attribute has to be registered under Bar before it can be used in a given module.

The exception to this rule is use:

defmodule Foo do
  defmacro __using__(_) do
    quote do
      import Foo
      Module.register_attribute(__MODULE__, :names, accumulate: true)
    end
  end

  defmacro add(name) do
    quote do
      @names unquote(name)
      :ok
    end
  end
end

defmodule Bar do
  use Foo

  add "john"
  add "henry"

  IO.inspect(@names) # This prints ["henry", "john"] right after the module is done compiling
end

This is how ExUnit accumulates test cases and uses @before_compile to inject a "run all test cases in test suite" function right before compilation, similar to something like this:

defmodule Calculator do
  def add(a, b), do: a + b
  def subtract(a, b), do: a - b
end

defmodule TestCase do
  defmacro __using__(_) do
    quote do
      import TestCase
      Module.register_attribute(__MODULE__, :tests, accumulate: true)
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(_env) do
    # Inject a run function into the test case after all tests have been accumulated
    quote do
      def run do
        Enum.each @tests, fn test_name ->
          result = apply(__MODULE__, test_name, [])
          state = if result, do: "pass", else: "fail"
          IO.puts "#{test_name} => #{state}"
        end
      end
    end
  end

  defmacro test(description, do: body) do
    test_name = String.to_atom(description)
    quote do
      @tests unquote(test_name)
      def unquote(test_name)(), do: unquote(body)
    end
  end
end

defmodule CalculatorTest do
  use TestCase
  import Calculator

  test "add 1, 2 should return 3" do
    add(1, 2) == 3
  end

  test "subtract 5, 2 should not return 4" do
    subtract(5, 2) == 4
  end
end

CalculatorTest.run
"add 1, 2 should return 3" => pass
"subtract 5, 2 should not return 4" => fail

Finally, we'll touch on deferring computation using macros.

Deferring Computation with Macros

Macros inject behavior into the callsite as-is and can be used to avoid immediate evaluation of an expression.

For instance:

defmodule Foo do
  def if?(condition, do: block, else: else_block) do
    case condition do
      true -> block
      false -> else_block
    end
  end
end

Foo.if? true do
  IO.puts("Truth")
else
  IO.puts("False")
end

"Truth"
"False"

Here, we tried implementing the new if using a regular function. However, the result is not what we expect — rather than only evaluating and printing "Truth", both "Truth" and "False" are printed.

This is because of the nature of a regular function: each block is evaluated immediately, so the case will not work.

If we use a macro instead, the macro has to be expanded first, generating an AST of the case first and evaluating the case accordingly. During this time, the condition is evaluated, before matching against the case, and, finally, the appropriate block is evaluated.

Note: This example is borrowed from Metaprogramming Elixir.

Macros in Elixir: A Powerful Tool, If Used Wisely

You can apply macros to many scenarios to extend an application's behavior in ways that normal code cannot.

However, macros are a double-edged sword — when misused, they can create confusion and muddy code's readability and semantic meaning.

In the final part of this metaprogramming series, we will delve into the common pitfalls you might encounter when working with macros in Elixir.

Thanks for reading, and see you next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Do Live Uploads in Phoenix LiveView

Roy Tomeij — 2021-10-12T00:00:00+00:00

The LiveView framework supports all of the most common features that Single-Page Apps must offer their users, including multipart uploads. In fact, LiveView can give us highly interactive file uploads, right out of the box.

In this post, we'll add a file upload feature to an existing Phoenix LiveView application. Along the way, you'll learn how to use LiveView to display upload progress and feedback while editing and saving uploaded files.

Setting Up the Image Upload Feature

The live upload examples that we'll be looking at in this post are drawn from the "Forms and Changesets" chapter in my book, Programming LiveView, co-authored with Bruce Tate. Check it out for an even deeper dive into LiveView forms and so much more.

In this example, we have an online game store — Arcade — that allows users to browse and review products. Admins can access a product management interface to create, edit, and delete the products we offer our users. We'll give admins the ability to upload a product image that is then stored in the database along with the given product. Let's plan out our new image upload feature before we start writing any code.

We'll begin in our application core by adding an image_upload field to the table and schema for products. Then, we'll create a LiveView form component that supports file uploads. Finally, we'll teach our component to report on upload progress and other bits of upload feedback.

This post will focus on adding the live upload functionality to an existing LiveView app that already implements:

A live view for displaying products.
A LiveView component that contains a form for creating/editing products.

We'll zero in on the code required to add the upload functionality to this form.

Check out my earlier post for a basic introduction to working with forms in LiveView.

Now we're ready to write some code.

Persist Product Images in Phoenix LiveView

Assuming our Phoenix LiveView app already has a products table and Product schema, we'll now update both to store an image_upload attribute. This attribute will point to the location of the uploaded file. Once we have our backend wired up, we'll be able to update the existing live view form to accommodate file uploads for a product.

We'll start at the database layer by generating a migration to add a field, :image_upload, to the products table.

First, generate your migration file:

[arcade] ➔ mix ecto.gen.migration add_image_to_products
* creating priv/repo/migrations/20201231152152_add_image_to_products.exs

This creates a migration file for us, priv/repo/migrations/20201231152152_add_image_to_products.exs. Open up that file and key in the contents to the change function:

defmodule Arcade.Repo.Migrations.AddImageToProducts do
  use Ecto.Migration

  def change do
    alter table(:products) do
      add :image_upload, :string
    end
  end
end

This code will add the new database field when we run the migration. Let's do that now:

[arcade] ➔ mix ecto.migrate

[info]  == Running 20201231152152 \

Arcade.Repo.Migrations.AddImageToProducts.change/0 forward
10:22:24.034 [info]  alter table products

10:22:24.041 [info]  == Migrated 20201231152152 in 0.0s

This migration added a new column — :image_upload — of type :string to the products table, but our schema still needs attention.

Update the corresponding Product schema by adding the new :image_upload field to the schema function, which should look like this:

defmodule Arcade.Catalog.Product do
  use Ecto.Schema
  import Ecto.Changeset

  schema "products" do
    field :description, :string
    field :name, :string
    field :sku, :integer
    field :unit_price, :float
    field :image_upload, :string
    timestamps()
  end

  @doc false
  def changeset(product, attrs) do
    product
    |> cast(attrs, [:name, :description, :unit_price, :sku, :image_upload])
    |> validate_required([:name, :description, :unit_price, :sku])
    |> validate_number(:unit_price, greater_than: 0)
    |> unique_constraint(:sku)
  end
end

Remember, the changeset cast/4 function must explicitly whitelist new fields, so make sure you add the :image_upload attribute there, as shown above.

Now that the changeset has an :image_upload attribute, we can save product records that know their image upload location. With that in place, we can make an image upload field available in the ProductLive.FormComponent's form. We're one step closer to giving users the ability to save products with images.

Now, let's turn our attention to the component.

How to Allow Live Uploads

We'll see our product changeset in action in a bit. First, we need to give the product form the ability to support file uploads. In our Phoenix application, both the "new product" and "edit product" pages use the ProductLive.FormComponent. This provides one centralized place to maintain our product form. Changes to this component will enable users to upload an image for a new product and a product they are editing.

To enable uploads for our component, or any live view, we need to call the allow_upload/3 function with a first argument of the socket. This will put the data into socket assigns that the LiveView framework will then use to perform file uploads. So, for a component, we'll call allow_upload/3 when the component first starts up and establishes its initial state in the update/2 function. For a live view, we'd call allow_upload/3 in the mount/3 function.

The allow_upload/3 function is a reducer that takes in an argument of the socket, the upload name, and the upload options and returns an annotated socket. Supported options include file types, file size, number of files per upload name, and more. Let's see it in action:

defmodule ArcadeWeb.ProductLive.FormComponent do
  use ArcadeWeb, :live_component
  alias Arcade.Product

  @impl true
  def update(%{product: product} = assigns, socket) do
    changeset = Product.changeset(product, %{})
    {:ok, socket
      |> assign(assigns)
      |> assign(:changeset, changeset)
      |> allow_upload(:image, accept: ~w(.jpg .jpeg .png), max_entries: 1)}
  end

In allow_upload/3, we pipe in a socket and specify a name for our upload: :image. We also provide some options — the maximum number of permitted files and the accepted file formats.

Let's take a look at what our socket assigns looks like after allow_upload/3 is invoked:

%{
  # ...
  uploads: %{
    __phoenix_refs_to_names__: %{"phx-FlZ_j-hPIdCQuQGG" => :image},
    image: #Phoenix.LiveView.UploadConfig<
      accept: ".jpg,.jpeg,.png",
      entries: [],
      errors: [],
      max_entries: 1,
      max_file_size: 8000000,
      name: :image,
      progress_event: #Function<1.71870957/3 ...>,
      ref: "phx-FlZ_j-hPIdCQuQGG",
      ...
    >
  }
}

The socket now contains an :uploads map that specifies the configuration for each upload field your live view allows. We allowed uploads for an upload called :image. So our map contains a key of :image pointing to a value of the configuration constructed using the options we gave allow_upload/3. This means that we can add a file upload field called :image to our form, and LiveView will track the progress of files uploaded via the field within socket.assigns.uploads.image.

You can call allow_upload/3 multiple times with different upload names, thus allowing any number of file uploads in a given live view or component. For example, you could have a form that allows a user to upload a main image, a thumbnail image, a hero image, and more.

Now that we've set up our uploads state, let's take a closer look at the :image upload configuration.

Upload Configurations in Phoenix LiveView

The :image upload config looks something like this:

#Phoenix.LiveView.UploadConfig<
  accept: ".jpg,.jpeg,.png",
  entries: [],
  errors: [],
  max_entries: 1,
  max_file_size: 8000000,
  name: :image,
  progress_event: #Function<1.71870957/3 ...>,
  ref: "phx-FlZ_j-hPIdCQuQGG",
  ...
>

Notice that it contains the configuration options we passed to allow_upload/3: the accepted file types list and file formats.

It also has an attribute called :entries, which points to an empty list. When a user uploads a file for the :image form field, LiveView will automatically update this list with the file upload entry as it completes.

Similarly, the :errors list starts out empty and is automatically populated by LiveView with any errors from an invalid file upload entry.

In this way, the LiveView framework does the work of performing the file upload and tracking its state for you. We'll see both of these attributes in action in a bit.

Now that we've allowed uploads in our component, we're ready to update the template with the file upload form field.

Render the File Upload Field

You'll use the Phoenix.LiveView.Helpers.live_file_input/2 function to generate the HTML for a file upload form field. Here's a look at our form component template:

<%= f = form_for @changeset, "#",
          id: "product-form",
          phx_target: @myself,
          phx_change: "validate",
          phx_submit: "save" %>

  <%= label f, :name %>
  <%= text_input f, :name %>
  <%= error_tag f, :name %>

  <%= label f, :description %>
  <%= text_input f, :description %>
  <%= error_tag f, :description %>

  <%= label f, :unit_price %>
  <%= number_input f, :unit_price, step: "any" %>
  <%= error_tag f, :unit_price %>

  <%= label f, :sku %>
  <%= number_input f, :sku %>
  <%= error_tag f, :sku %>

  <% # File upload fields here: %>
  <%= label f, :image %>
  <%= live_file_input @uploads.image %>

  <%= submit "Save", phx_disable_with: "Saving..." %>
</form>

Notice the use of live_file_input/2 with an argument of @uploads.image. Remember, socket.assigns has a map of uploads. Here, we provide @uploads.image to live_file_input/2 to create a form field with the right configuration and tie that form field to the correct part of socket state. This means that LiveView will update socket.assigns.uploads.image with any new entries or errors that occur when a user uploads a file via this form input.

The live view can present upload progress by displaying data from @uploads.image.entries and @uploads.image.errors. LiveView will handle all of the details of uploading the file and updating socket assigns @uploads.image entries and errors for us. All we have to do is render the data that is stored in the socket. We'll take that on soon.

Now, we should be able to see the file upload field displayed in the browser like this:

And if you inspect the element, you'll see that the live_file_input/2 function generated the appropriate HTML:

You can see that the generated HTML has the accept=".jpg,.jpeg,.png" attribute set, thanks to the options we passed to allow_upload/3.

LiveView also supports drag-and-drop file uploads. All we have to do is add an element to the page with the phx-drop-target attribute, like this:

<%= f = form_for @changeset, "#",
          id: "product-form",
          phx_target: @myself,
          phx_change: "validate",
          phx_submit: "save" %>

  # ...

  <div phx-drop-target="<%= @uploads.image.ref %>">
    <%= live_file_input @uploads.image %>
  </div

  # ...
</form>

We give the attribute a value of @uploads.image.ref. This socket assignment is the ID that LiveView JavaScript uses to identify the file upload form field and tie it to the correct key in socket.assigns.uploads. So now, if a user clicks the "choose file" button or drags-and-drops a file into the area of this div, LiveView will store the file info in the socket.assigns.uploads assignment, under the name of the specified upload, in that upload's :entries list.

As with other form interactions, LiveView automatically handles the client/server communication. When the user submits the form, LiveView's JavaScript will first upload the file(s) and then invoke the handle_event/3 callback for the form's phx-submit event. To process the file upload, this event handler will need to consume the file upload stored in socket.assigns.uploads.image.entries. Let's do that now.

Consume Uploaded Entries

Our handle_event/3 function for the phx_submit: "save" form event will use LiveView's consume_uploaded_entry/3 function to process the uploaded file. For now, we'll have our function write the uploaded file to our app's static assets in priv/static/images. This is so that we can display it on the product show template later on.

Here's our code:

defmodule ArcadeWeb.ProductLive.FormComponent do
# ...
  def handle_event("save", product_params, socket) do
    file_path =
      consume_uploaded_entry(socket, :image, fn %{path: path}, _entry ->
        dest = Path.join("priv/static/uploads", Path.basename(path))
        File.cp!(path, dest)
        Routes.static_path(socket, "/uploads/#{Path.basename(dest)}")
      end)

    product = save_product(Map.put(product_params, :image_upload, file_path)

    {:noreply,
      socket
      |> push_redirect(to: Routes.product_show_path(socket, :show, product))}
  end
end

We save the image to static assets and return the file path from consume_uploaded_entry/3. Then, we call a helper function — save_product/1 (not pictured here) — to update the product with the given form params, including the new :image_upload attribute set to our new file path. Finally, we redirect to the Product Show page.

To see our code in action, let's add some markup to the product show template to display image uploads. Then, we'll try out our feature.

Display Image Uploads

Open up lib/arcade_web/live/product_live/show.html.leex and add the following markup to display the uploaded image or a fallback:

<article class="column">
   <img
      alt="product image" width="200" height="200"
      src="<%=Routes.static_path(
              @socket,
              @product.image_upload || "/images/default-thumbnail.jpg")%>">
  </article>
<!-- product details... -->

Perfect. Now, we can test drive it. Visit /products/1/edit and upload a file:

Once you submit the form, you'll see the show page render the newly uploaded image, like this:

We did it! The LiveView framework handled all of the client/server communication details that make the page interactive. LiveView performed the file upload for you and made responding to upload events easy and customizable. You only needed to tell the live view which uploads to track and what to do with uploaded files when the form is submitted. Then you added the file upload form field to the page with the view helper and LiveView handled the rest!

There's one last thing to do. Earlier, I promised reactive file uploads that share feedback with the user. Let's take a look at this now.

Display Upload Feedback in Phoenix LiveView Forms

We know that LiveView automatically updates the :entries and :errors lists in the uploads config portion of socket.assigns once the upload begins. Let's display this information in the template to give the user real-time progress tracking. The code is amazingly simple. We'll iterate over @uploads.image.entries to display the progress for each entry:

<%= for entry <- @uploads.image.entries do %>
  <p>
    <progress value={entry.progress} max="100"> <%= entry.progress %>% </progress>
  </p>
<% end %>

Here, we use the HTML progress tag to create a simple progress bar that is populated with the progress of our file upload in real-time. As LiveView's JavaScript is uploading the file for you, LiveView is updating the value of the entry's progress in socket assigns. This causes the relevant portion of the template to re-render, thereby showing the updated progress bar in real-time. LiveView handles the work of tracking the changes to the image entry's progress. All we have to do is display it.

You can use a similar approach to iterate over and display any errors stored in @uploads.image.errors. You won't have to do any work to validate files and populate errors. LiveView handles those details. All you need to do is display any errors based on the needs of your user interface. Here's a look at the code:

<%= for err <- upload_errors(@uploads.image, entry) do %>
  <p class="alert alert-danger"><%= friendly_error(err) %></p>
<% end %>

Here, we use the Phoenix.LiveView.Helpers.upload_errors/2 function to return the errors for the specified upload.

The error messages aren't very user-friendly, though. So, we'll implement a helper function, friendly_error/1, in our LiveView component that looks like this:

defmodule ArcadeWeb.ProductLive.FormComponent do
  # ...

  def friendly_error(:too_large), do: "Image too large"
  def friendly_error(:too_many_files), do: "Too many files"
  def friendly_error(:not_accepted), do: "Unacceptable file type"
end

There's more that LiveView file uploads can do. LiveView makes it easy to:

cancel an upload
upload multiple files for a given upload config
upload files directly from the client to a cloud provider

and more.

Check out the LiveView file upload documentation for details.

Wrap-up: Build Complex Forms Easily with Phoenix LiveView

LiveView enables reactive file uploads right out of the box. Without writing any JavaScript, or even any custom HTML, you can build interactive file upload forms directly into your live view.

LiveView handles the details of client/server communication and upload state management, leaving you on the hook to write a very small amount of custom code specifying how your uploads should behave and how uploaded files should be saved.

This is a pattern you'll see again and again in LiveView — the framework handles the communication and state management details of our SPA, and we can focus on writing application-specific code to support our features.

Now that you've had a glimpse of what LiveView can do with form uploads, you're ready to build complex, interactive forms that support real-time uploads in the wild. Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Under the Hood of Macros in Elixir

Roy Tomeij — 2021-10-05T00:00:00+00:00

This post was updated on 9 August 2023 with the sections 'What are Macros in Elixir?' and 'Why You Need Macros'.

Welcome back to part two of this series on metaprogramming in Elixir. In part one, we introduced metaprogramming and gave a brief overview of macros.

In this part, we will explore the inner workings and behaviors of macros in more depth.

Although we had discussed what macros are in the previous post, it doesn't hurt to refresh our memory on what macros are and why they are important.

What are Macros in Elixir?

Macros are powerful compile-time constructs that generate and modify code before it is compiled into bytecode. Elixir macros are used to define new language constructs, extend already existing ones, or perform other code transformations that are necessary for developer productivity.

Why You Need Macros

In a nutshell, Elixir macros are important because:

They can help you handle abstraction. By writing macros to handle all sorts of specialized functionalities in your app, you'll likely end up with a more maintable code base as your app grows, since complexity is handled properly.
You can use them to run compile-time code checks, which will help your overall code quality.
They are perfect for creating Elixir libraries. If you've used a library like Cowboy or even Ecto, then what you see is an example of an Elixir macro. So if your goal is to create an Elixir library to share with the Elixir community or for your own purposes, you will find that macros are the perfect way to go about it.

And with that, we can now see what happens during the compilation process of an Elixir app.

Stages of Elixir's Compilation Process

We can boil down the Elixir compilation process to the following basic stages:

(Note: the actual compilation process of an Elixir program is more intricate than above.)

The compilation process can be broken down into the following phases:

Parsing — The Elixir source code (program) is parsed into an AST, which we will call the initial AST.
Expansion — The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a final AST is generated.
Bytecode generation phase — After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode.

As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work.

Expansion Phase

Let's start by first examining the expansion phase on a general level.

The compiler will expand macros (as per Macro.expand) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro until it reaches its most fundamental AST form.

As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time.

If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively.

We can observe this behavior as follows:

defmodule Foo do
  defmacro foo do
    quote do
      IO.inspect("hello")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}
iex(3)> ast |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]},
    :inspect
  ]}, [], ["hello"]}

ast represents the initial AST generated by the compiler before expansion. It holds a reference to the macro Foo.foo but it is not expanded as the macro has not been evaluated yet.

When we call Macro.expand on the given AST, the compiler begins by injecting the behavior of the macro into the callsite. We can expand the AST one step at a time using Macro.expand_once.

Contexts in Macros

Now that we understand the basics of the expansion phase, we can investigate the parts of a macro.

Macros contain two contexts — a macro context and a caller context:

defmacro foo do
  # This is the macro's context, this is executed when the macro is called

  # This is the return value of the macro (AST)
  quote do
    # This is the caller's context, this is executed when the callsite is called
  end
end

As you can see, the macro's context is any expression declared before the quote.

The caller's context is the behavior declared in the quote. The quote generated AST is the macro's output and is injected into and expanded at the callsite.

The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from Metaprogramming Elixir, illustrates this:

defmodule Mod do
  defmacro definfo do
    IO.puts "definfo :: Macro's context #{__MODULE__}"

    quote do
      IO.puts "definfo :: Caller's context #{__MODULE__}"

      def friendly_info do
        IO.puts "friend_info :: Module name #{__MODULE__}"
      end
    end
  end
end

defmodule MyModule do
  require Mod
  Mod.definfo
end

iex(1)> c "context.exs"
definfo :: Macro's context Elixir.Mod
definfo :: Caller's context Elixir.MyModule
[Mod, MyModule]
iex(2)> MyModule.friendly_info
friend_info :: Module name Elixir.MyModule
:ok

As you can see, the module that executes the macro's context is Mod. But the module that executes the caller's context is MyModule — the callsite where the macro is injected and expanded.

Similarly, when we declare friendly_info, we inject this function into the callsite of the macro, which is MyModule. So the function now 'belongs' to MyModule.

But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another?

Order of Evaluation in Macro and Caller Contexts

The key difference between the macro context and the caller context is that behavior is evaluated at different times.

Let's look at an example:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")
    quote do
      IO.puts("caller")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}

iex(3)> ast |> Macro.expand(__ENV__)
macro
{{:., [],
  [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]},
 [], ["caller"]}

When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite.

However, we can go a level deeper.

Let's look again at the first component of the expansion phase: Macros are executed. Their output (AST) is injected and expanded into the callsite.

For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase.

If we think of macros as regular functions, the macro context is the function body and the caller context is the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the quote is evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the macro.

The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module.

The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated.

The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate?

Well, we can use another example here:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")

    quote do
      IO.puts("caller")
    end
  end
end

defmodule Bar do
  require Foo

  def execute do
    IO.puts("execute")
    Foo.foo
  end
end

macro
iex(1)> Bar.execute
execute
caller

The caller context evaluates when the function is called (as we established earlier).

However, something interesting happens with our macro context — it evaluates when the module compiles. This evaluation only happens once when Bar compiles, as evidenced by the lack of "macro" in our output when we call Bar.execute. Why is this the case?

Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called.

This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts.

We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code.

We use the caller context when we want to invoke behavior injected into the callsite at evaluation.

Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit unquote.

Revisiting `unquote`

In part one of this series, we established that unquote evaluates a given expression and injects the result (as an AST) into the AST built from quote. This is only a piece of the puzzle.

Let's dig deeper to understand the behavior of unquote during compilation and the necessity of using it.

While the rest of the quote body is evaluated at the same time as the callsite, unquote is evaluated (immediately) during compile-time — when the macro is evaluated. unquote aims to evaluate and inject the result of a given expression. This expression might contain information that is only available during the macro evaluation, including variables that are initialized during this process. unquote must be evaluated during compile-time along with the macro, so that the AST of the result injects into the quote that we build.

But why do we need to unquote the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using unquote against one that does not:

defmodule Foo do
  defmacro foo(x) do
    quote do
      IO.inspect(x)
    end
  end
end

defmodule Bar do
  defmacro bar(y) do
    quote do
      IO.inspect(unquote(y))
    end
  end
end

iex(1)> require Foo
iex(2)> require Bar
iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3)
iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3)
iex(5)> ast_foo |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]},
    :inspect
  ]}, [], [{:x, [counter: -576460752303423448], Foo}]}

iex(6)> ast_bar |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]},
    :inspect
  ]}, [],
 [
   {:+, [context: Elixir, import: Kernel],
    [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}
 ]}

Observe that the expanded Foo.foo AST is vastly different from the Bar.bar AST even though they are both given the same variable. This is because Elixir is quite literal with variable references. If a variable is referenced without unquote, an AST of that variable reference injects into the AST.

Using unquote ensures that the underlying AST of the variable's value injects into the quote body.

Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter?

The scoping of variables in macros can be a confusing subject, so let's demystify it.

Variable Scoping in Macros

Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in macros, and when to use the options unquote and bind_quoted in quote.

Due to function clause scoping, the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function. Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros.

When the macro context is evaluated, its arguments and any initialized variables are 'in scope.' This is why unquote can evaluate variable references declared as arguments of the macro or any variables initialized in the macro context.

Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution.

To understand this difference better, let's look at a few examples:

defmacro foo do
  quote do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

In this first example, unquote will not work. The variable x has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of unquote runs too early, so we cannot reference our variable x when we need to. When unquote evaluates during compile-time, it attempts to evaluate the variable reference expression of x and finds that it is not in scope.

How can we fix this? By disabling unquoting. This means disabling the immediate evaluation of unquote. We only want unquote to evaluate when our caller context evaluates. This ensures that unquote can properly reference a variable in scope (x) as variable initialization would have occurred during the evaluation of the callsite.

defmacro foo do
  quote unquote: false do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

This example highlights the impact of scoping in macros. If we attempt to access a variable that is available during the evaluation of the macro context, unquote as-is is perfect for us.

However, suppose we try to access a variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior to initialize variables in scope before unquote attempts to reference them.

Let's apply this understanding to two other examples.

defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def bar, do: IO.inspect(unquote(x))
  end
end

In this example, we have initialized the variable x from a keyword list. As the keyword list is initialized during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to:

Generate an initialization of the variable during the evaluation of the callsite, and
Disable unquoting behavior.

We have to bind opts to the caller context, as the variable is no longer in scope during the evaluation of the callsite.

Finally, we have:

defmacro foo(x) do
  quote do
    def bar, do: IO.inspect(unquote(x))
  end
end

In this last example, x remains a variable in scope during the evaluation of the macro context — i.e. when the macro is called. The immediate evaluation of unquote works in our favor. It renders unquote(x) valid, as x is in scope when unquote is evaluated.

Macro Hygiene in Elixir

While we are on the topic of scopes in macros, let's discuss macro hygiene.

According to tutorialspoint.dev:

Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.

This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables.

Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the official tutorial.

A macro variable is declared within the body of quote, while a caller variable is declared within the callsite of the macro.

defmodule Foo do
  defmacro change do
    quote do: a = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 1

In this example, a referenced in change is not the same variable a in the scope of go. Even when we attempt to change the value of a, the value of a in the scope of go remains untouched.

However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope. Elixir provides the macro var! to bridge the gap between these two scopes:

defmodule Foo do
  defmacro change do
    quote do: var!(a) = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 13

This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to).

You can apply the same hygiene to aliases and imports.

Understanding `require`

In the code examples shown in this section, we have always used require <module> before we invoke the macros within the module. Why is that?

This is the perfect segue into how the compiler resolves modules containing macros — particularly the order in which modules are compiled.

In Elixir, modules compile in parallel, and usually — for regular modules — the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations.

However, as macros must be available during compile-time, the module these macros belong to must be compiled beforehand. Here's where require comes into the picture. require explicitly informs the compiler to compile and load the module containing the macro first.

We can use an example to illustrate this behavior:

defmodule Foo do
  IO.puts("Compiling Foo")

  defmacro foo do
    IO.puts("Foo.foo := macro")
    quote do
      IO.puts("Foo.foo := caller")
    end
  end
end

defmodule Bar do
  IO.puts("Compiling Bar")

  require Foo

  IO.puts("Bar := before Foo.foo")
  Foo.foo()
end

Compiling Foo
Foo.foo := macro
Compiling Bar
Bar := before Foo.foo
Foo.foo := caller

(Note that this is just an approximation of the actual compilation process, but it aims to paint a clearer picture of how require works.)

Let's try to understand why the outputs are in this order.

Firstly, Bar tries to compile. The compiler scans and finds a require for Foo before evaluating any module-level expressions within the module (such as IO.puts). So it pauses the compilation of Bar and compiles the Foo module first. As Foo is compiled, module-level code — like IO.puts — is evaluated, and the compiler prints the first line of the output.

Once Foo is compiled, the compiler returns to Bar to resume compilation. Bar is parsed, macro calls are executed, and the macro context is evaluated. Even though Foo.foo is called after IO.puts("Bar := before Foo.foo"), the evaluation of the macro call takes precedence over the evaluation of module-level code.

During expansion, Foo.foo's caller context is injected and expanded into the callsite in Bar. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration.

In essence, require instructs the compiler on the order of compilation that each module should go through if there are macro dependencies. This ensures that the macros are available during compile-time.

Escaping Macros in Elixir

Before explaining what Macro.escape does, let's look at an example:

iex(1)> x = {1, 2, 3}
iex(2)> quote do: unquote(x)
{1, 2, 3}
iex(3)> ast = quote do: IO.inspect(unquote(x))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]}
iex(4)> Code.eval_quoted(ast)
** (CompileError) nofile: invalid quoted expression: {1, 2, 3}

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the
AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3

That is a strange error. Based on our understanding of unquote and macros, the code should work as intended, but it doesn't. Why is that?

Well, the answer is found on iex(2). When we attempt to unquote x, we return not an AST, but a tuple — the same one initially assigned to x. The error then points to the fact that the tuple is an invalid quoted expression.

When we unquote(x) as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a valid AST and throws an error.

So, how do we fix it?

We need to convert the raw tuple referenced by x into a valid AST. This can be achieved by escaping this value using Macro.escape. Let's understand what Macro.escape does:

iex(1)> a = {1, 2, 3}
{1, 2, 3}
iex(2)> Macro.escape(a)
{:{}, [], [1, 2, 3]}
iex(3)> quote do: unquote(a)
{1, 2, 3}
iex(4)> quote do: unquote(Macro.escape(a))
{:{}, [], [1, 2, 3]}

In iex(2), we see that Macro.escape(a) returns an AST of the tuple, not the raw tuple — and this is exactly what we are looking for. By combining Macro.escape's behavior with unquote, we can inject the AST of the tuple into the quote as seen in iex(4).

Let's test this:

iex(1)> x = {1, 2, 3}
{1, 2, 3}
iex(2)> quote do: unquote(Macro.escape(x))
{:{}, [], [1, 2, 3]}
iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x)))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [],
 [{:{}, [], [1, 2, 3]}]}
iex(4)> Code.eval_quoted(ast)
{1, 2, 3}
{{1, 2, 3}, []}

As you can see, the code works just as intended because we escape the tuple.

Often when working with data structures like tuples and dictionaries, you may find that the injected data from unquote does not inject a valid AST. In these cases, you should use Macro.escape before unquote.

Guard Clauses and Pattern Matching

Finally, it's worth mentioning that, much like regular functions defined through def, macros can use guard clauses and pattern matching:

defmacro foo(x) when x == 4 do
  IO.inspect("macro with x being 4")
end

defmacro foo(_) do
  IO.inspect("macro with any other value")
end

Up Next: Applications of Macros in Elixir

Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally.

In part three, we will look at the many applications of macros in Elixir.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Real-Time Form Validation with Phoenix LiveView

Roy Tomeij — 2021-09-28T00:00:00+00:00

LiveView is a compelling choice for building modern web apps. Built on top of Elixir's OTP tooling and leveraging WebSockets, it offers super-fast real-time, interactive features alongside impressive developer productivity.

LiveView keeps the developer's mind firmly rooted on the server-side, even when testing and debugging. This can empower you to deliver interactive features in single-page apps faster than ever before. Some of the most common interactions on the web are form validation and submission. These days, users expect to see form feedback presented to them in real-time, and LiveView offers first-class support for exactly that.

In this post, I'll show you how to build LiveView forms that validate changes and provide feedback to the user in real-time. Along the way, you'll learn how to model change in your Phoenix application with schemaless changesets and compose LiveView code to handle that change.

The Feature: Adding a Form to a Phoenix LiveView App

The form examples we'll be looking at in this post are inspired by the "Forms and Changesets" chapter in my book, Programming LiveView, co-authored with Bruce Tate. Check it out for an even deeper dive into LiveView testing and so much more.

We'll focus on that last piece of functionality. A user can invite a friend to play a game with them by filling out a form with the friend's email address. This will email a link to the recipient's email address so they can join the game.

Our form will need to implement some validation — namely, to ensure that a valid email address is provided. It should show any validation errors, as well as the results of a successful form submission in real-time. We won't worry too much about the exact details of sending emails for now. Instead, we'll keep our focus on the form validations in LiveView.

We'll begin where you'll almost always want to start when building a new feature in a Phoenix application — in the application core. We'll model game invitations in their module. You'll build a boundary layer in a Phoenix context module that we'll call on in LiveView later to validate form input and send invitation emails.

Let's get started!

Model Change in Phoenix with Ecto Changesets

We're almost ready to start writing code. But first, let's think about the role that forms and changesets play in our Phoenix application.

Consider Ecto changesets. Changesets are policies for changing data, and they:

Cast unstructured user data into a known, structured form — most commonly, an Ecto database schema, ensuring data safety.
Capture differences between safe, consistent data and a proposed change, allowing for efficiency.
Validate data using known consistent rules, ensuring data consistency.
Provide a contract for communicating error states and valid states, ensuring a common interface for change.

def changeset(game, attrs) do
  game
  |> cast(attrs, [:name, :description, :unit_price, :sku])
  |> validate_required([:name, :description, :unit_price, :sku])
  |> unique_constraint(:sku)
end

The changeset/2 function captures differences between the structured game and the unstructured attrs.

Then, with cast/4, the changeset trims the attributes to a known field list and converts them to the correct types. This ensures safety by guaranteeing that you won't let any unknown or invalid attributes into your database.

Finally, the validate/2 and unique_constraint/2 functions validate the inbound data, ensuring consistency.

The result is a data structure with known states and error message formats, ensuring interface compatibility.

Consequently, any forms that use this changeset know exactly how to behave — validating form input and presenting errors in accordance with the changeset's rules.

In this post, we're going to shift off of this well-known path of generated changesets. You'll see just how versatile changesets can be when it comes to modeling changes to data, with or without a database. You'll build a custom, schemaless changeset for data that isn't backed by a database table, and you'll use that changeset in a form within a live view.

Schemaless Changesets in the Phoenix Application Core

You've likely used changesets to model changes to data that is persisted in your database. Our game invitation feature doesn't require database persistence, however. A user will provide the email address of the friend they'd like to invite, and our application will simply send out the email with the link to the game. We don't need to store that invitee's data at this point.

Luckily, we can use schemaless changesets to model data that doesn't get saved to the database. A schemaless changeset is based on a simple Elixir map or struct, rather than an Ecto schema-backed struct. The only difference is that, when working with a plain Elixir struct, we need to provide the changeset with the type of information that Elixir would normally handle. We'll see this in action in a moment.

First, let's define the core module that we'll use to model a game Recipient. Create a new file, arcade/lib/game/Recipient.ex, and define the module like this:

defmodule Arcade.Invite.Recipient do
  defstruct [:name, :email]
end

Our module is simple so far. It implements a struct with two keys, :name and :email.

Next up, we need to give our module awareness of the types that will be considered valid by any changeset we create. Let's use a module attribute to store this map of types so that we can access it later:

defmodule Arcade.Invite.Recipient do
  defstruct [:name, :email]
  @types %{game_name: :string, email: :string}
end

Now, we'll alias the module and import Ecto.Changeset so that we can use the changeset functions:

defmodule Arcade.Invite.Recipient do
  defstruct [:name, :email]
  @types %{game_name: :string, email: :string}

  alias Arcade.Invite.Recipient
  import Ecto.Changeset
end

Finally, we're ready to define the changeset/2 function that will be responsible for casting recipient data into a changeset and validating it:

defmodule Arcade.Invite.Recipient do
  # ...
  def changeset(%Recipient{} = invitation, attrs) do
    {invitation, @types}
    |> cast(attrs, Map.keys(@types))
    |> validate_required([:game_name, :email])
    |> validate_format(:email, ~r/@/)
  end
end

We validate the presence of the :game_name and :email attributes, and then validate the format of :email.

Now, we can create recipient changesets like this:

iex> alias Arcade.Invite.Recipient
iex> i = %Recipient
iex> Recipient.changeset(r, %{email: "juniper@email.com", game_name: "Chess"})
#Ecto.Changeset<...valid?: true>

Let's see what happens if we try to create a changeset with an attribute of an invalid type:

iex> Recipient.changeset(r, %{email: "juniper@email.com", game_name: 1234})
#Ecto.Changeset<errors: [game_name: {"is invalid", ...]}],valid?: false>

Ecto.Changeset.cast/4 relies on @types to identify the invalid type and provide a descriptive error.

Next, try a changeset that breaks one of the custom validation rules:

iex> Recipient.changeset(r, %{email: "juniper's email", game_name: "Chess"})
#Ecto.Changeset<changes: %{email: "juniper's email", ...},
  errors: [email: {"has invalid format", ...}],valid?: false>

This function successfully captures our change policy in code, and the returned changeset tells the user exactly what is wrong.

Now that our changeset is up and running, let's quickly build out an Invite context that will present the interface for interacting with the changeset.

The Boundary Layer in Elixir

Create a file, lib/arcade/invite.ex and add in the following:

defmodule Arcade.Invite do
  alias Arcade.Invite.Recipient

  def change_invitation(%Recipient{} = recipient, attrs \\ %{}) do
    Recipient.changeset(recipient, attrs)
  end

  def send_invite(recipient, attrs) do
    # send email to promo recipient
  end
end

This context is a beautifully concise boundary for our service. The change_invitation/2 function returns a recipient changeset, and send_invite/2 is a placeholder for sending a game invite email.

Other than the internal tweaks we made inside Recipient.changeset/2, building a Phoenix context module with a schemaless changeset looks identical to building an Ecto-backed one. When all is said and done, in the view layer, schemaless changesets and schema-backed ones will look identical.

Let's turn our attention to the view layer now and build out our live view.

Build and Define the Live View

This live view will have the feel of a typical live view with a form. First, we'll create a simple route and wire it to a live view. Next, we'll use our Invite context to produce a schemaless changeset, and add it to the socket within a mount/3 function. We'll render a form with this changeset and apply changes to the changeset by handling events from the form.

Let's get going!

Create a file, lib/arcade_web/live/invite_live.ex and fill in the following:

defmodule ArcadeWeb.InviteLive do
  use ArcadeWeb, :live_view
  alias Arcade.Invite
  alias Arcade.Invite.Recipient

  def mount(_params, _session, socket) do
    {:ok, socket}
  end
end

We pull in the LiveView behavior, alias our modules for later use and implement a simple mount/3 function.

Let's use an implicit render/1. Create a template file in lib/arcade_web/live/invite_live.html.leex, starting with some simple markup:

<h2>Invite a Friend to Play!</h2>
<h4>
  Enter the name of a game and your friend's email below and we'll send them an
  invite to join you in playing a game!
</h4>

Now, let's define a live route and fire up the server. In the router, add the following route behind authentication:

scope "/", ArcadeWeb do
  pipe_through [:browser, :require_authenticated_user]
  live "/invite", InviteLive
end

With that, you should be able to fire up the Phoenix server, point your browser at /invite and see your live view render the invitation template:

As the live view is up and running, we're ready to build out the form for an invitation recipient.

Render the Phoenix LiveView Form

We'll use mount/3 to store a recipient struct and a changeset in the socket:

 def mount(_params, _session, socket) do
  {:ok,
    socket
    |> assign_recipient()
    |> assign_changeset()}
end

def assign_recipient(socket) do
  socket
  |> assign(:recipient, %Recipient{})
end

def assign_changeset(%{assigns: %{recipient: recipient}} = socket) do
  socket
  |> assign(:changeset, Invite.change_invitation(recipient))
end

The mount/3 function uses two helper functions, assign_recipient/1 and assign_changeset/1, to add a recipient struct and a changeset for that recipient to socket assigns. These pure, single-purpose reducer functions are reusable building blocks for managing the live view's state.

Remarkably, the schemaless changeset can be used in our form exactly like database-backed ones. We'll use socket.assigns.changeset in the template's form, like this:

<%= f = form_for @changeset, "#",
  id: "invite-form",
  phx_change: "validate",
  phx_submit: "save" %>

  <%= label f, :game_name %>
  <%= text_input f, :game_name  %>
  <%= error_tag f, :game_name %>

  <%= label f, :email %>
  <%= text_input f, :email%>
  <%= error_tag f, :email %>

  <%= submit "Send Invite"%>
</form>

Now, if you point your browser at /invite, you should see this:

Our form implements two LiveView bindings, phx-change, and phx-submit. Let's dig into these events now.

Handle Form Events in LiveView

We'll start with a look at the phx-change event. LiveView will send a "validate" event each time the form changes and include the form params in the event metadata.

So, we'll implement a handle_event/3 function for this event that builds a new changeset from the params and adds it to the socket:

# lib/arcade_web/live/invite_live.ex
def handle_event(
      "validate",
      %{"recipient" => recipient_params},
      %{assigns: %{recipient: recipient}} = socket) do
  changeset =
    recipient
    |> Invite.change_invitation(recipient_params)
    |> Map.put(:action, :validate)

    {:noreply,
    socket
    |> assign(:changeset, changeset)}
end

Let's break this down. The Invite.change_invitation/2 context function creates a new changeset using the recipient from socket state and the params from the form change event.

Then, we use Map.put(:action, :validate) to add the validate action to the changeset, a signal that instructs Phoenix to display errors. Phoenix will not display the changeset's errors otherwise.

When you think about it, this approach makes sense. Not all invalid changesets should show errors on the page. For example, the empty form for the new changeset shouldn't show any errors because the user hasn't provided any input yet. So, the Phoenix form_for function needs to be told when to display a changeset's errors. If the changeset's action is empty, then no errors are set on the form object — even if the changeset is invalid and has a non-empty :errors value.

Finally, assigns/2 adds the new changeset to the socket, triggering render/1 and displaying any errors.

Let's take a look at the form tag that displays those errors on the page. Typically, each field has a label, an input control, and an error tag, like this:

<%= label f, :email %>
<%= text_input f, :email%>
<%= error_tag f, :email %>

The error_tag/2 Phoenix view helper function displays the form's errors for a given field on a changeset, when the changeset's action is :validate.

Now, point your browser at /invite and fill out the form with a game name and an invalid email. As you can see in this image, the UI updates to display the validation errors:

That was surprisingly easy! We built a simple and powerful live view with a reactive form that displays any errors in real-time.

The live view calls on the context to create a changeset, renders it in a form, validates it on form change, and then re-renders the template after each form event. We get reactive form validation for free without writing any JavaScript or HTML. We let Ecto changesets handle the data validation rules, and we let the LiveView framework handle the client/server communication for triggering validation events and displaying the results.

There's just one thing about our form validation that needs some improvement. You'll notice that as soon as you start typing into the email form, an error appears because our validation event fires whenever the form field changes. This doesn't present our users with the best experience — we're telling them there is an error with their input before they get a chance to finish typing their full email.

Instead, we want the validation event to fire when a user clicks away or blurs from the email input. Luckily for us, LiveView makes it easy to implement this functionality with the help of the phx-debounce binding. Update your email form field to look like this:

<%= text_input f, :email, phx_debounce: "blur" %>

Now the "validate" event will only fire when a user blurs away from the email input field, and we won't show any premature validation error messages.

Learn more about LiveView's support for debouncing and other rate-limiting options.

As you might imagine, the phx-submit event works very similarly to phx-change. The "save" event fires when the user submits the form. To respond to this event, we can implement a handle_event/3 function that uses the (currently empty) context function, Invite.send_invite/2. The context function should create and validate a changeset.

If the changeset is, in fact, valid, we can pipe it to some helper function or service that handles the details of sending invitation emails.

If the changeset is not valid, we can return an error tuple. Then we can update the UI with a success or failure message accordingly. I'll leave you to practice what you're learning by implementing this behavior on your own.

Wrap-up: LiveView — A Great Choice to Build and Validate Forms in Elixir

Now you've seen that Ecto changesets are not tightly coupled to the database. Schemaless changesets let you tie backend services to Phoenix forms any time you require validation and security, whether or not your application needs to access a full relational database.

LiveView supports custom integration of forms to backend code with these schemaless changesets. To do so, you need only provide the Changeset.cast/4 function with the first argument of a two-tuple holding both data and type information. This type of code is ideal for implementing form scenarios requiring validation but without the typical database backend.

Whether you're working with schema-backed or schemaless changesets, LiveView provides real-time form validation and feedback, with very little hand-rolled code. We can use LiveView event bindings to handle form validation and submission in real-time with a few simple event handlers that call out to our nice, clean Phoenix context code.

With that, you have everything you need to build basic forms in LiveView. To dig deeper into LiveView's rich forms offerings, check out the docs. Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Application Code Upgrades in Elixir

Roy Tomeij — 2021-09-14T00:00:00+00:00

In this third and final part of this series, we will look at what happens during an application upgrade.

Let's get going!

Set-up: Create an Appup File in Elixir

As with a single module, we need new compiled code for a fresh version of an application.

But an application can consist of many modules and have running processes. So we need a scenario to specify what to upgrade and how to do it.

These scenarios are called application upgrade files (.appup files).

Let's try to make an appup file for our application. Imagine that we want to upgrade our counters so that we can specify counter increment speed.

Prepare a New Version of the Application in Elixir

First, we'll add our application to git and tag the old version:

git add .
git commit -m 'initial commit'
git tag 'v0.1.0'

Now we should make a new version.

In mix.exs, update the version to 0.1.1:

...
 def project do
    [
      app: :our_new_app,
      version: "0.1.1",
...

We want interval to be an external parameter, not a module attribute. Make the following changes in lib/our_new_app/counter.ex:

defmodule OurNewApp.Counter do
  ...
  def start_link({start_from, interval}) do
    GenServer.start_link(__MODULE__, {start_from, interval})
  end

  ...

  def init({start_from, interval}) do
    Process.flag(:trap_exit, true)

    st = %{
      current: start_from,
      timer: :erlang.start_timer(interval, self(), :tick),
      terminator: nil,
      interval: interval
    }

    {:ok, st}
  end

  ...

  def handle_info({:timeout, _timer_ref, :tick}, st) do
    :erlang.cancel_timer(st.timer)

    new_current = st.current + 1

    if st.terminator && rem(new_current, 10) == 0 do
      # we are terminating
      GenServer.reply(st.terminator, :ok)
      {:stop, :normal, %{st | current: new_current, timer: nil}}
    else
      new_timer = :erlang.start_timer(st.interval, self(), :tick)
      {:noreply, %{st | current: new_current, timer: new_timer}}
    end
  end

  ...
end

Now add a code_change callback to the same file:

defmodule OurNewApp.Counter do
  ...

  def code_change(_old_vsn, st, new_interval) do
    {:ok, Map.put(st, :interval, new_interval)}
  end
end

And change supervisor specs in lib/our_new_app/counter_sup.ex:

  @impl true
  def init(start_numbers) do
    children =
      for start_number <- start_numbers do
        # We can't just use `{OurNewApp.Counter, start_number}`
        # because we need different id's for children

        Supervisor.child_spec({OurNewApp.Counter, {start_number, 200}}, id: start_number)
      end

    Supervisor.init(children, strategy: :one_for_one)
  end

Let's construct our_new_app.appup file. We need to update our supervision specs and pass a new tick interval (250) to change_code callback:

{
    "0.1.1",
    [{"0.1.0", [
        {update, 'Elixir.OurNewApp.CounterSup', supervisor},
        {update, 'Elixir.OurNewApp.Counter', {advanced, 250}}
    ]}],
    [{"0.1.0", []}]
}.

Note that this file is written in Erlang syntax.

Now tag your new version:

git add .
git commit -m 'added customizable intervals'
git tag 'v0.1.1'

Try to upgrade 0.1.0 to 0.1.1.

Checkout, compile, and run 0.1.0 version:

git checkout v0.1.0
iex -S mix

In a separate shell and different directory, checkout the new version and put the appup file in the appropriate place:

...
git checkout v0.1.0
mix compile
cp our_new_app.appup _build/dev/lib/our_new_app/ebin

Run the Application Upgrade in Elixir

We are ready to upgrade the application. In the running iex session, do the following:

iex(1)> Application.spec(:our_new_app)
[
  description: 'our_new_app',
  id: [],
  vsn: '0.1.0',
  modules: [OurNewApp, OurNewApp.Application, OurNewApp.Counter,
   OurNewApp.CounterSup],
  maxP: :infinity,
  maxT: :infinity,
  registered: [],
  included_applications: [],
  applications: [:kernel, :stdlib, :elixir, :logger],
  mod: {OurNewApp.Application, []},
  start_phases: :undefined
]

You can see that the old application is running.

Run an "orphaned" counter outside the supervision tree (we will need it later):

iex(2)> {:ok, pid} = OurNewApp.Counter.start_link(30000)
{:ok, #PID<0.147.0>}

Check that your appup file is correct and the OTP knows how to upgrade your application:

iex(3)> :release_handler.upgrade_script(:our_new_app, '/path/to/new/version/of/our_new_app/_build/dev/lib/our_new_app/')
{:ok, '0.1.1',
 [
   {:load_object_code,
    {:our_new_app, '0.1.1', [OurNewApp.CounterSup, OurNewApp.Counter]}},
   :point_of_no_return,
   {:suspend, [OurNewApp.CounterSup]},
   {:load, {OurNewApp.CounterSup, :brutal_purge, :brutal_purge}},
   {:code_change, :up, [{OurNewApp.CounterSup, []}]},
   {:resume, [OurNewApp.CounterSup]},
   {:suspend, [OurNewApp.Counter]},
   {:load, {OurNewApp.Counter, :brutal_purge, :brutal_purge}},
   {:code_change, :up, [{OurNewApp.Counter, 250}]},
   {:resume, [OurNewApp.Counter]}
 ]}

Check your counter processes and their pids:

iex(4)> Supervisor.which_children(OurNewApp.CounterSup)
[
  {20000, #PID<0.143.0>, :worker, [OurNewApp.Counter]},
  {10000, #PID<0.142.0>, :worker, [OurNewApp.Counter]}
]

Upgrade the application!

iex(5)> :release_handler.upgrade_app(:our_new_app, '/path/to/new/version/of/our_new_app/_build/dev/lib/our_new_app/')
{:ok, []}
iex(6)>
02:46:48.286 [info]  terminating with {{:badkey, :interval, %{current: 33478, terminator: nil, timer: #Reference<0.948322908.3672375303.146224>}}, [{OurNewApp.Counter, :handle_info, 2, [file: 'lib/our_new_app/counter.ex', line: 52]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 680]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 756]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}, counter is 33478

02:46:48.291 [error] GenServer #PID<0.147.0> terminating
...

Let's see what happened:

iex(1)> Application.spec(:our_new_app)
[
  description: 'our_new_app',
  id: [],
  vsn: '0.1.1',
  modules: [OurNewApp, OurNewApp.Application, OurNewApp.Counter,
   OurNewApp.CounterSup],
  maxP: :infinity,
  maxT: :infinity,
  registered: [],
  included_applications: [],
  applications: [:kernel, :stdlib, :elixir, :logger],
  mod: {OurNewApp.Application, []},
  start_phases: :undefined
]
iex(2)> Supervisor.which_children(OurNewApp.CounterSup)
[
  {20000, #PID<0.143.0>, :worker, [OurNewApp.Counter]},
  {10000, #PID<0.142.0>, :worker, [OurNewApp.Counter]}
]
iex(3)> pids = for {_, pid, _, _} <- Supervisor.which_children(OurNewApp.CounterSup), do: pid
[#PID<0.143.0>, #PID<0.142.0>]
iex(4)> OurNewApp.Counter.get(Enum.at(pids, 0))
27225
iex(5)> OurNewApp.Counter.get(Enum.at(pids, 1))
17235
iex(6)> :sys.get_state(Enum.at(pids, 0))
%{
  current: 27468,
  interval: 250,
  terminator: nil,
  timer: #Reference<0.948322908.3672375311.146315>
}
iex(7)> :sys.get_state(Enum.at(pids, 1))
%{
  current: 17476,
  interval: 250,
  terminator: nil,
  timer: #Reference<0.948322908.3672375311.146330>
}
iex(8)> :sys.get_state(OurNewApp.CounterSup)
{:state, {:local, OurNewApp.CounterSup}, :one_for_one,
 {[20000, 10000],
  %{
    10000 => {:child, #PID<0.142.0>, 10000,
     {OurNewApp.Counter, :start_link, [{10000, 200}]}, :permanent, 5000,
     :worker, [OurNewApp.Counter]},
    20000 => {:child, #PID<0.143.0>, 20000,
     {OurNewApp.Counter, :start_link, [{20000, 200}]}, :permanent, 5000,
     :worker, [OurNewApp.Counter]}
  }}, :undefined, 3, 5, [], 0, OurNewApp.CounterSup, [10000, 20000]}

Our upgrade process was successfully completed.

You can see that:

The new version of the application is now running.
Our child counters are alive — they updated their state and are functioning.
From the internal state of OurNewApp.CounterSup, we see those child specifications for our counters updated too! Now, if they die, they will properly restart.

But what about the error GenServer #PID<0.147.0> terminating?

Recall that #PID<0.147.0> is the pid of our "orphaned" counter, which was running outside of the supervision tree. As the application upgrade process traverses the supervision tree and updates processes, the "orphaned" counter's state was not updated. But the code of the OurNewApp.Counter did update, so the "orphaned" counter process died: new code met its old state.

We've seen how to upgrade a single running application. We needed only two special tools for that: an .appup file and :release_handler.upgrade_app/2 function. It was also crucial for us to follow OTP principles.

Wrap-up and What to Learn Next

I hope you've enjoyed this whirlwind ride through production code upgrades in Elixir! We started with my guide to hot code reloading, followed by the best use of supervisors when building applications.

This final article has demonstrated how following OTP principles can show us the way to powerful application code upgrades.

It's worth noting that the application code upgrade I've demonstrated here still has disadvantages. The critical issue is that if the whole OS beam process restarts, this will load our application's old code (unless we take some action).

How can we handle this potential problem, I hear you ask? With so-called release upgrades. This awesome article from 'Learn you some Erlang' is a good starting point to dive into release upgrades.

Thanks again for taking the time to read this series — and happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Metaprogramming in Elixir

Roy Tomeij — 2021-09-07T00:00:00+00:00

This post was updated on 9 August 2023. The 'Types of Metaprogramming' and 'Why Do Metaprogramming?' sections were switched, so that 'Why Do Metaprogramming?' is now covered before 'Types of Metaprogramming'.

In this world, there are many mysteries — but few are as elusive as metaprogramming in Elixir.

In this four-part series, we'll start by looking at core concepts and then explore how metaprogramming operates in Elixir specifically.

Let's develop an understanding of metaprogramming and uncover some Elixir metaprogramming secrets!

Introducing Metaprogramming in Elixir

According to Harald Sondergaard, metaprogramming is:

a programming technique in which computer programs have the ability to treat other programs as their data; meaning that a program can be designed to read, generate, analyze, or transform other programs, and even modify itself while running.

In essence, metaprogramming — much like metadata — revolves around "a set of programs that describe and give information about other programs" (adapted from the Oxford dictionary definition of metadata).

Before we get into how to do metaprogramming in Elixir, let's understand why we are doing it in the first place.

Why Do Metaprogramming?

We will look at how to use metaprogramming specifically in Elixir. But first, let's cover some general concepts and benefits of metaprogramming to understand why we do it in the first place:

Adapt code to runtime factors - One of the benefits of metaprogramming is generating code that can adapt to different run-time factors. This is great if you need your software to be dynamic and respond to varying factors at runtime.
Performance optimization - It's possible to use metaprogramming to customize your code in such a way that it leads to better performance compared to trying to achieve the same with a static codebase.
Reduce errors - Using metaprogramming technniques, you can reduce errors that are bound to occur when you copy-paste code a lot. By using macros that generate code on the fly, it's possible to reduce the need for manual copy-paste operations that would likely introduce errors and other code mistakes at compile-time.
DSLs - With metaprogramming, a developer can easily create a domain-specific language to better express how to handle a particular scenario in their project.

This is just a small snapshot of the possibilities available to a developer through using metaprogramming (we'll also highlight a few uses later in the article). That said, one thing you'll observe here is the frequent use of the terms "run-time" and "compile-time". Let's see what they mean next.

Defining Run-time and Compile-time

We can broadly classify metaprogramming into two categories: compile-time and run-time metaprogramming.

But what exactly are run-time and compile-time? They both refer to stages of a program's life cycle.

Compile-time is the stage at which source code converts to binary code or intermediate binary code for a machine or virtual machine to execute. Run-time refers to when code executes.

The program life cycle includes the following steps:

Source: https://en.wikipedia.org/wiki/Program_lifecycle_phase

Note that this is not a complete representation of the entire program life cycle, just a simplified one.

Compilation "sets the program in stone" by converting it into binary code. Metaprogramming exposes this process to allow developers to "move computation from run-time to compile-time" or "generate code using compile-time computations". This essentially allows the modification of the source code before/during compile-time, meaning that the generated binary code is slightly different.

Self-modifying code is rather unique. In essence, it performs reflection during run-time. The life cycle of a self-modifying program looks a little different:

Note: You can substitute the "binary" for any intermediate language generated by the compiler, such as JVM bytecode or, in Elixir's case, BEAM VM bytecode.

Types of Metaprogramming

There are two types of metaprogramming that prescribe varying degrees of control over a given program:

Introspection

Introspection refers to a program revealing metadata about other programs or itself.

This definition broadly covers the first part of the definition of metaprogramming: "a program can be designed to read...[and] analyze...other programs". The program has access to information about itself or other programs.

Reflection

Reflection refers to a program modifying other programs or itself.

If a program can modify other programs or itself, it — by definition — has access to the metadata of the program, revealing information like names of functions.

By looking at the two types of metaprogramming, we can conclude that reflection encompasses introspection, or introspection is a subset of reflection:

Interesting Uses of Metaprogramming

The general definition of metaprogramming also encompasses the tools used in a program's life cycle.

For instance, a language compiler is a metaprogramming application designed to receive another program as input and generate binary as output.

We can narrow down broad metaprogramming use cases to the following, more specific, applications in a programming language context:

Code generation

By generating code dynamically during compile-time, it's available during run-time. When the nature of the code that's generated is not fixed, this can prove especially useful. For instance, you can use code generation to design domain-specific languages (DSLs) or generate functions based on input such as files or APIs.

Code instrumentation

Code instrumentation refers to the measure of a program's performance, error diagnosis, and logging of trace information.

Metaprogramming enables this through dynamic program analysis — software analysis performed by running software through a real or virtual processor.

Code instrumentation enables features like code coverage, memory error detection, fault localization, and concurrency errors.

Behavioral changes

This refers to changing the behavior of a program through metaprogramming. Behavioral changes can include feature toggling, where a given feature is toggled on/off through a flag that is read during compile-time/run-time.

This article series is about metaprogramming within Elixir, so our key focus will be on code generation.

Metaprogramming in Elixir: The Basics

Elixir applies a style of metaprogramming known as macro system metaprogramming (also used in other languages like Rust and Lisp).

In Elixir, metaprogramming allows developers to leverage existing features to build new features that suit their individual business requirements.

The foundation of metaprogramming in Elixir is macros.

Defining Macros

According to the official macros documentation:

Macros are compile-time constructs that are invoked with Elixir's AST as input and a superset of Elixir's AST as output.

There are two critical components to this definition. Let's break them down:

Compile-time constructs - evaluated and available during compile-time
Elixir's AST - Abstract Syntax Trees (ASTs) are tree representations of the abstract syntax structure of the source code

We use the representations of the source code as building blocks for compile-time constructs. Since the compiler reasons with the source code through ASTs, we effectively "speak" the compiler's language to build constructs that it can directly reason with.

In Elixir, ASTs are tuples, so we reason with the compiler in a manner that is familiar to us. We do not need to deviate from Elixir's syntax to begin writing macros, which lowers our barrier to entry of learning macros. On top of that, we do not even need to write ASTs ourselves. There are constructs in Elixir to handle all of that heavy lifting for us.

The above definition also mentions how a macro receives an AST as input and returns a superset of AST as output. So, you can think of a macro as a regular function with inputs, behavior, and an output. The overall goal is to use a given AST to generate a new AST for the compiler to use.

There is more to come on the compilation process of Elixir programs in part two of this series.

Starting Small with Macros

Now that we understand macros, let's dip our toes into the water and implement a basic macro.

We'll start with a very basic comparison of a macro to a regular function.

The Elixir documentation inspires this code example:

defmodule Foo do
  defmacro macro_inspect(value) do
    IO.inspect(value)
    value
  end

  def func_inspect(value) do
    IO.inspect(value)
    value
  end
end

To define a macro, we use defmacro and declare the parameters just as we would a regular function.

Running the macro in IEX yields the following results:

iex(1)> import Foo
iex(2)> macro_inspect(1 + 2)
{:+, [context: Elixir, import: Kernel], [1, 2]}
3
iex(3)> func_inspect(1 + 2)
3
3

Observe that rather than printing the result of 1 + 2, the macro prints a tuple instead (the AST as input that we defined earlier).

When a macro is first declared, the arguments of that macro are automatically converted into AST so that you don't need to parse the arguments manually. The arguments will not be evaluated beforehand.

However, when the value of the macro is returned, it yields the result of 1 + 2. The macro should return an AST as output (and it is). However, this AST as output is compiled and executed once the macro is called. The expression 1 + 2 is evaluated first, then returned.

Once we understand the basic syntax and declaration of a macro, we can explore the structure of the AST.

AST Structure

As mentioned earlier, the AST is the representation of the source code as a syntax tree. In the example above, we inspect the AST of the expression 1 + 2.

We can break down the AST structure into three components:

Atom — representing the name of the operation
Metadata of the expression
Arguments of the operation

{
  :+,                                 # operation name,
  [context: Elixir, import: Kernel],  # metadata,
  [1, 2]                              # operation arguments
}

While you must understand what comprises an AST, we rarely need to read/write raw ASTs.

Elixir makes it ridiculously easy to interface with macros, so we hardly even need to think about the structure of the AST that we are working on — everything is handled for us.

Interacting with ASTs

As mentioned earlier, ASTs represent the source code and are the input and output of macros. They are the cornerstone of macros. We need to interact with the AST representations of expressions freely, without getting bogged down by reading and writing the ASTs ourselves.

This is where quote and unquote come into the picture.

To generate the AST representation of an expression or body, we use quote:

quote do
  1 + 2 * 3
end

{:+, [context: Elixir, import: Kernel],
 [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}

When we use quote, we build an AST. While the example above is relatively simple, we will soon discover that quote can be used to build much more complex ASTs.

What if we have a value we want to use in our quote, such as the arguments? We attempt to introduce an external (outside of quote) variable into quote, by using unquote.

unquote evaluates its argument, which is an expression, and injects the result (as an AST) into the AST being built. As everything in Elixir is an expression, we evaluate expressions to inject the results.

For instance, if unquote receives a variable, we will evaluate that expression as the underlying expression referenced by the variable and inject that.

If unquote receives a full expression like 1 + 2 * 3, we will evaluate that to 7 and inject that. unquote expects that the result of the expression is a valid AST.

In part two of this series, we'll discuss the consequences of having an invalid AST and delve into macros more deeply.

Do you recall that macros automatically convert arguments into their AST forms? We will leverage that behavior:

defmodule Foo do
  defmacro foo(exp) do
    quote do
      doubled = unquote(exp) * 2
      doubled
    end
  end
end

Foo.foo(1 + 2 * 3)
14

As you can see, we have built a macro called foo which receives an expression as an argument. Then, we begin to build an AST for the macro in quote. We use unquote(exp) to inject the value of the exp argument into the AST.

You might ask yourself: How do I know that the expression is injected and not evaluated right away?

Well, we can use a handy tool to inspect the AST of the macro and understand how it works under the hood:

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo(1 + 2 * 3)
iex(3)> ast |> Macro.expand(__ENV__)
{:__block__, [],
 [
   {:=, [],
    [
      {:doubled, [counter: -576460752303423358], Foo},
      {:*, [context: Foo, import: Kernel],
       [
         {:+, [context: Elixir, import: Kernel],
          [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]},
         2
       ]}
    ]},
   {:doubled, [counter: -576460752303423358], Foo}
 ]}

First, we generate the AST of the macro call and assign it to a variable.

Then, with our ast variable, we will use Macro.expand to expand the AST to its fullest form.

We'll look at macro expansion next time. For now, think of it as peeling back the layers of an AST to its most fundamental components.

As you can see, the expanded form of the Foo.foo call contains the AST of 1 + 2 * 3. This proves that unquote only injected the AST of the expression into the quote AST, but didn't evaluate it. The evaluation is performed later on (we will get into this in part two as well).

Note: Macro.expand will only attempt to perform expansion on the root node of the AST.

You can find more information about Macro.expand in the docs.

`quote` Options in Elixir

Now that we understand the fundamentals of macros, we can start to look at our quote options.

While there are several options with quote, we will focus on the three most frequently used and introduce the concepts behind each option.

unquote

Toggles the unquoting behavior in quote. By disabling it, any unquote call is converted to an AST of the macro call (as with any other macro/function call).

This defers the evaluation of unquote to a later point. I'll explain why you'd want to do so in the next part of this series.

For now, let's look at the following example:

iex(1)> a = [foo: 1, bar: 1]
iex(2)> ast = quote do: unquote(a)
  [foo: 1, bar: 1]
iex(3)> ast = quote unquote: false, do: unquote(a)
  {:unquote, [], [{:a, [], Elixir}]}

When we leave the unquoting behavior enabled (iex(2)), unquote(a) will evaluate a as an expression. This returns the keyword list, which is then injected into the quote AST — and the result is as expected.

However, when we disable the unquoting behavior (iex(3)), unquote(a) is converted into another AST expression, which is injected into the quote AST as-is.

bind_quoted

Disables unquoting behavior in the quote and binds given variables in the body of quote.

Binding moves the variable initialization into the body of quote.

We can observe this behavior using Macro.to_string:

iex(1)> a = [foo: 1, bar: 2]
iex(2)> ast = quote bind_quoted: [a: a], do: IO.inspect(a)
iex(3)> ast |> Macro.to_string |> IO.puts
(
  a = [foo: 1, bar: 2]
  IO.inspect(a)
  :ok
)
:ok

As you can see, bind_quoted adds a "copy" of a into the body of quote by assigning it in the body of quote.

In a macro, this is equivalent to binding the variable to the caller context, as the variable is initialized during the evaluation of the callsite.

Note: Contexts will be discussed in greater detail next time.

  defmodule Foo do
    defmacro foo(x) do
      quote bind_quoted: [x: x] do
        IO.inspect(x)
        end
    end
end

location

This option controls whether run-time errors from a macro are reported from the caller or inside the quote.

By setting this option to :keep, error messages report specific lines in the macro that cause the error, rather than the line of the callsite.

You can see a code example in the docs.

Build a Simple Macro in Elixir

We should now be able to build a simple macro that mimics the behavior of an if statement.

Recall that an if statement is comprised of the following components:

if (condition) do
  # body
else
  # body
end

We can replicate this structure using our own macro:

defmodule NewIf do
  defmacro if?(condition, do: block, else: other) do
    quote do
      cond do
        unquote(condition) == true -> unquote(block)
        unquote(condition) == false -> unquote(other)
      end
    end
  end
end

iex(1)> require NewIf
iex(2)> NewIf.if? 4 == 5, do: :yes, else: :no
:no

This macro can receive three arguments:

condition - predicate to evaluate if? statement against
do - block to execute when condition is true
else - block to execute when condition is false

In Elixir, such blocks can be declared as arguments if they follow the following syntax: <formal name>: <variable name>. The formal name is the name used when you call the macro. The variable name is the name used in the macro when you're attempting to reference the block.

After receiving these three arguments, we start by building an AST using quote.

Using a cond statement, we determine which body if? should execute. We use unquote to inject the values of condition, block, and other into the AST we are building.

In doing so, when the macro is evaluated, the condition is evaluated to be true/false, and, based on that result, we will either execute block or other.

We wrap up this behavior into an AST returned by quote (which is the return value of the macro).

Next Up: Macros in Detail

Now we have a good grasp on the foundations of metaprogramming in general and specifically in Elixir.

Join me for the next part of this series, where we'll look into the intricacies behind macros and how everything works.

Until next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

LiveView Integration Tests in Elixir

Roy Tomeij — 2021-08-31T00:00:00+00:00

In the second part of this two-part series, we'll write an integration test that validates interactions within a single live view, and an integration test that validates the interactions between two separate live views.

You will focus on testing the behavior of the survey results chart filter from the previous post. We'll use the LiveViewTest module's functions to simulate LiveView connections without a browser. With the help of this module, your tests can mount and render live views, trigger events, and then execute assertions against the rendered view.

That's the whole LiveView lifecycle.

So let's get going and write some interactive LiveView tests!

Testing Interactions within a Live View

Our first integration test focuses on the interactions within a single live view. We'll validate the live view's behavior when a user performs some activity on the page. First off, we're going to take a look at the feature that we'll be testing.

The Feature

The SurveyResultsLive component mentioned in the previous post is rendered within a parent live view, AdminDashboardLive, that lives at the /admin-dashboard route. Here's a refresher of the content displayed by that component:

Here, you see a chart that displays survey results and can be filtered by a given age group.

Our test will simulate a user's visit to /admin-dashboard, followed by their filter selection of the 18 and under age group. The test will verify an updated survey results chart that displays product ratings from users in that age group.

Because components run in their parent's processes, we'll focus our tests on the AdminDashboardLive view. LiveViewTest helper functions will run our admin dashboard live view and interact with the survey results chart. Along the way, you'll get a taste for the wide variety of interactions that the LiveViewTest module allows you to test.

Begin by setting up a LiveView test for the AdminDashboardLive view.

The Test

It's best to segregate unit tests and integration tests into their own modules, so create a new file test/gamestore_web/live/admin_dashboard_live_test.exs and define the module with some fixtures, like this:

defmodule GamestoreWeb.AdminDashboardLiveTest do
  use GamestoreWeb.ConnCase

  import Phoenix.LiveViewTest
  alias Gamestore.{Accounts, Survey, Catalog}

  @create_product_attrs %{description: "test description", name: "Test Game", sku: 42, unit_price: 120.5}
  @create_user_attrs %{email: "test@test.com", password: "passwordpassword"}
  @create_user2_attrs %{email: "test2@test.com", password: "passwordpassword"}
  @create_user3_attrs %{email: "test3@test.com", password: "passwordpassword"}
  @create_demographic_attrs %{gender: "female", year_of_birth: DateTime.utc_now.year - 15}
  @create_demographic_over_18_attrs %{gender: "female", year_of_birth: DateTime.utc_now.year - 30}

  defp product_fixture do
    {:ok, product} = Catalog.create_product(@create_product_attrs)
    product
  end

  defp user_fixture(attrs \\ @create_user_attrs) do
    {:ok, user} = Accounts.register_user(attrs)
    user
  end

  defp demographic_fixture(user, attrs) do
    attrs =
      attrs
      |> Map.merge(%{user_id: user.id})
    {:ok, demographic} = Survey.create_demographic(attrs)
    demographic
  end

  defp rating_fixture(user, product, stars) do
    {:ok, rating} = Survey.create_rating(%{stars: stars, user_id: user.id, product_id: product.id})
    rating
  end

  defp create_product(_) do
    product = product_fixture()
    %{product: product}
  end

  defp create_user(_) do
    user = user_fixture()
    %{user: user}
  end

  defp create_demographic(user, attrs \\ @create_demographic_attrs) do
    demographic = demographic_fixture(user, attrs)
    %{demographic: demographic}
  end

  defp create_rating(user, product, stars) do
    rating = rating_fixture(user, product, stars)
    %{rating: rating}
  end
end

Let's break this down. First, our test module uses the GamestoreWeb.ConnCase behavior. This lets us route to live views using the test connection by giving our tests access to a context map with a key of :conn pointing to a value of the test connection. Then, import the LiveViewTest module to get access to LiveView testing functions.

Lastly, define some fixtures that you'll use to create test data. The nitty-gritty details of those fixture functions aren't important. Just understand that they generate the database records needed to log in users and render the admin dashboard page to display a chart with product ratings.

Now that our module is set up, we'll add a describe block to encapsulate the feature we're testing—the survey results chart functionality:

describe "Survey Results" do
  setup [:register_and_log_in_user, :create_product, :create_user]

  setup %{user: user, product: product} do
    create_demographic(user)
    create_rating(user, product, 2)

    user2 = user_fixture(@create_user2_attrs)
    create_demographic(user2, @create_demographic_over_18_attrs)
    create_rating(user2, product, 3)
    :ok
  end
  # test coming soon!
end

Two calls to setup/1 seed the test database with a product, users, demographics, and ratings. One of the two users is in the 18 and under age group, and the other is in a different age group. Then we create a rating for each user.

We're also using a test helper provided for us by the Phoenix authentication generator—register_and_log_in_user/1. This function creates a conn struct with a logged-in user, a necessary step because visiting the /admin-dashboard route requires an authenticated user.

Now that our setup is complete, define the test:

describe "Survey Results" do
  # ...
  test "it filters by age group", %{conn: conn} do
  end
end

Before writing the body of our test, make a plan. To test this feature, we need to:

Mount and render the live view.
Find the age group drop-down menu and select an item from it.
Assert that the re-rendered survey results chart has the correct data and markup.

This is the pattern you'll always apply to testing live view features: run the live view, simulate some interaction, then validate the rendered result. This pattern should sound familiar—it neatly matches up to the three-step testing process we've been using so far:

Set up preconditions
Provide input
Validate your expectations

Begin with the first step: mounting and rendering the LiveView. Call the LiveViewTest.live/2 function, which takes in the test conn and spawns a simulated live view process:

test "it filters by age group", %{conn: conn} do
  {:ok, view, _html} = live(conn, "/admin-dashboard")
end

The call to live/2 returns a three-element tuple with :ok, the LiveView process, and the rendered HTML returned from the live view's call to render/1. We don't need to access that HTML in this test, so ignore it.

Components run in their parent's process. That means the test must start up the AdminDashboardLive view rather than rendering just the SurveyResultsLive component. By spawning the AdminDashboardLive view, we're also rendering the components of the view.

So, by interacting with the view variable representing the AdminDashboardLive process above, we'll interact with elements within the SurveyResultsLive component and test that it behaves appropriately in response to events. This is the correct way to test LiveView component behavior within a live view page.

The test has a running live view now, so provide your input by selecting the 18 and under age filter. This will require two steps:

Find the age group drop-down menu
Choose an item from it

Use the LiveViewTest.element/3 function to find the age group drop-down on the page.

Assuming the drop-down menu HTML form element has an ID of age-group-filter, you can target it with element/3 like this:

test "it filters by age group", %{conn: conn} do
  {:ok, view, _html} = live(conn, "/admin-dashboard")
  html =
    view
    |> element("#age-group-form")
end

element/3 returns Phoenix.LiveViewTest.Element struct that we can pipe into another LiveViewTest function in order to simulate the selection of an item and submission of the form:

test "it filters by age group", %{conn: conn} do
  {:ok, view, _html} = live(conn, "/admin-dashboard")
  html =
    view
    |> element("#age-group-form")
    |> render_change(%{"age_group_filter" => "18 and under"})
end

The LiveViewTest.render_change/2 function is one of the functions you'll use to simulate user interactions when testing live views. It takes an argument of the selected element and some params, triggering a phx-change event. Here, make sure to call render_change/2 with the exact params that would be sent to the live view when the user selects an age group filter in the UI.

This event will trigger the associated handler, invoking the reducers that update our socket, and re-rendering the survey results chart with the filtered product rating info.

With our setup and input in place, we're ready to write our assertions. The call to render_change/2 will return the re-rendered template. Add an assertion that the re-rendered chart displays the correct data by validating the presence of an updated title for the product's average rating:

test "it filters by age group", %{conn: conn} do
  {:ok, view, _html} = live(conn, "/admin-dashboard")
    view
    |> element("#age-group-form")
    |> render_change(%{"age_group_filter" => "18 and under"})
    |> assert =~ "<title>2.00</title>" # validates that we now display an average rating of 2.00
end

Once again, the details of our assertion aren't important. Just understand that when the product ratings are filtered by age group, you can expect to see the element on the page: <title>2.00</title>.

And with that, our first integration test is complete! The LiveViewTest module provided everything we needed to mount and render a connected live view, target elements within that live view—even elements nested within child components—and assert the state of the view after firing DOM events against those elements.

The test code is clean and elegantly composed with a simple pipeline. All of it is written in Elixir with ExUnit and LiveViewTest functions—we didn't need to bring in any JavaScript dependencies. As a result, writing our test was a straightforward process. We ended up with a reliable test that runs fast and is easy to read.

This is only a small subset of the LiveViewTest functions that support LiveView testing, but there are many more LiveViewTest functions that allow you to send any number of DOM events—blurs, form submissions, live navigation, and more. Learn more about them in the docs.

Before we go, let's write one more integration test; this time, to exercise interactions between live views.

Testing Distributed Updates in LiveView

Testing message passing in a distributed application can be painful, but LiveViewTest makes it easy to test the PubSub-backed real-time features that you can build into your live views. That is because LiveView tests interact with views via process communication. Since PubSub uses simple Elixir message passing, it's easy to test a live view's ability to handle such messages: use send/2.

In this section, we'll write an integration test that validates the behavior of the AdminDashboardLive when it receives a specific message over PubSub.

The Feature

Our AdminDashboardLive supports the following real-time update feature: when a user anywhere in the world submits a new product rating, then the survey results chart on the admin dashboard updates accordingly, in real-time.

The following code flow backs this:

When a user submits a product rating, then a PubSub event, "rating_created" is broadcast over a topic.
The AdminDashboardLive view subscribes to that topic and responds to the event by re-rendering the SurveyResultsLive component with updated data from the database.

The details of the code aren't important for our purposes today—a high-level understanding is all that's needed to write our test. Let's get started.

The Test

Like in our unit test earlier, you can group similar test cases in a single describe block. We already have a describe block in our integration test module for "Survey Results". Since the test of the real-time update feature also describes the behavior of the SurveyResultsLiveComponent, we'll add another test case to this same describe block:

describe "Survey Results" do
  # ...
  test "it updates to display newly created ratings", %{conn: conn, product: product} do
end

This time, our test case retrieves both the test conn and the product from the setup context. Use this product to create a new rating for display.

Once again, before filling in the body of our test, make a plan. Follow the same three-step process you used for your earlier integration test:

Mount and render the connected live view.
Interact with that live view—in this case, by sending the "rating_created" message to the live view.
Re-render the view and verify changes in the resulting HTML.

Start by mounting and rendering the live view with the live/2 function:

test "it updates to display newly created ratings", %{conn: conn, product: product}
  {:ok, view, html} = live(conn, "/admin-dashboard")
  assert html =~ "<title>2.50</title>"
end

Here, we add an intermediate assertion to check the starting state of the product ratings label. Expect to change this value once you create a new rating and send the "rating_created" message to the live view.

Next up, provide your input (a two-step process). First, create a new rating for the product:

test "it updates to display newly created ratings", %{conn: conn, product: product}
  {:ok, view, html} = live(conn, "/admin-dashboard")
  assert html =~ "<title>2.50</title>"

  # create a new user + demographic and then create a new rating with those records
  user3 = user_fixture(@create_user3_attrs)
  create_demographic(user3)
  create_rating(user3, product, 3)
end

Then, send the "rating_created" message to the live view:

test "it updates to display newly created ratings", %{conn: conn, product: product}
  {:ok, view, html} = live(conn, "/admin-dashboard")
  assert html =~ "<title>2.50</title>"

  # create a new user + demographic and then create a new rating with those records
  user3 = user_fixture(@create_user3_attrs)
  create_demographic(user3)
  create_rating(user3, product, 3)

  # send the message to the live view
  send(view.pid, %{event: "rating_created"})
end

Here, use a simple send/2 to mimic the code flow of PubSub broadcasting the "rating_creating" message. Our live view should respond by re-rendering the SurveyResultsLive component with fresh data from the DB, including the newly created rating. All we need to do now is add our assertion:

test "it updates to display newly created ratings", %{conn: conn, product: product}
  {:ok, view, html} = live(conn, "/admin-dashboard")
  assert html =~ "<title>2.50</title>"

  # create a new user + demographic and then create a new rating with those records
  user3 = user_fixture(@create_user3_attrs)
  create_demographic(user3)
  create_rating(user3, product, 3)
  # send the message to the live view
  send(view.pid, %{event: "rating_created"})
  # give the live view time to re-render
  :timer.sleep(2)
  assert render(view) =~ "<title>2.67</title>"
end

And that's it! You:

Established your initial state by mounting and rendering the live view with the call to live/2
Provided some input by creating a new rating record and sending a message to the live view with send/2
Validated your expectations by asserting that the re-rendered view had some expected content.

Our three-step LiveView testing procedure neatly applies to both integration tests that exercise internal live view behavior, and tests that validate the interactions between live view processes.

Now for wrapping up.

Wrap Up: Write Robust and Comprehensive LiveView Tests

LiveView empowers you to write robust and comprehensive tests without a huge investment of engineering effort.

In the previous post, we comprised our individual live views from pipelines of single-purpose reducers. This provided opportunities for deep unit testing to quickly cover lots of scenarios and edge cases. You can even use the same elegant reducer pipelines in your tests to verify the behavior of your live view pipelines.

This article showed that the LiveViewTest module provides all the functionality you need to exercise the full range of LiveView interactions in integration tests. We can use the functions in the LiveViewTest module to apply the same three-step process that guides all of our tests, making it quick and easy to spin up tests for even complex LiveView interactions.

LiveView is built on top of OTP, and a live view is nothing more than a process. This means you can easily test interactions between live views by relying on simple message passing.

The powerful set of tools in your LiveView testing kit is just one of the many reasons that teams can be so productive in LiveView. You and your team can guarantee comprehensive test coverage for your LiveView applications, ensuring that you move quickly while maintaining bug-free code.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Using Supervisors to Organize Your Elixir Application

Roy Tomeij — 2021-08-23T00:00:00+00:00

In the previous chapter of this series, we looked at hot code reloading in Elixir and why we should use GenServer to implement long-running processes.

But to organize a whole application, we need one more building block — supervisors. Let's take a look at supervisors in detail.

Defining an OTP Application

According to the documentation:

In OTP, application denotes a component implementing some specific functionality, that can be started and stopped as a unit, and that can be reused in other systems. This module interacts with application controller, a process started at every Erlang runtime system.

This module contains functions for controlling applications (for example, starting and stopping applications), and functions to access information about applications (for example, configuration parameters).

In other words, an application is a kind of package that contains reusable modules, has name, version, specific dependencies, etc.

Mix creates a new application when we run:

mix new our_new_app

But there is one crucial difference that distinguishes OTP applications from packages in other languages. OTP applications can be started and stopped and have their own running entities.

You can usually create such applications in Elixir with the command:

mix new our_new_app --sup
cd our_new_app

This creates an additional file for us: lib/our_new_app/application.ex. It implements the so-called application behavior. Its primary purpose is to implement start/2 function, which should start a supervision tree.

What Are Supervision Trees in Elixir?

So what is a supervision tree? I use the following analogy: a running OTP system is like a whole OS with its own lightweight processes. They start, work, and terminate. As in a real OS, we need a tool that helps us:

Start the system in the correct order
Handle abnormal situations when a process dies due to some errors
Stop the system correctly.

In a real OS, we have Systemd (on some Linux OSes) or launchd on MacOS. In OTP, there are supervisors and Supervisor module.

We can organize our processes in the following way using supervisors:

Leaf processes in this scheme are generally GenServer or similar processes.

As in Systemd, if a process fails, we can choose to do nothing. Another option is to restart the process over and over again until it completes normally, or together with sibling processes.

Earlier, in Erlang, it was tricky to build supervision trees, but Elixir helps us a lot with this.

It's also worth noting that there is some good documentation available about supervisors.

Connecting GenServers to a Supervision Tree

Let's again look at what mix created for us in lib/our_new_app/application.ex:

  def start(_type, _args) do
    children = [
      # Starts a worker by calling: OurNewApp.Worker.start_link(arg)
      # {OurNewApp.Worker, arg}
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: OurNewApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

It starts a supervisor and clearly shows how to run our worker as a child.

Let's do that. We will be able to increment a given number periodically and report its state on demand in our sample process.

First, create a GenServer in lib/our_new_app/counter.ex:

defmodule OurNewApp.Counter do
  use GenServer
  require Logger

  @interval 100

  def start_link(start_from, opts \\ []) do
    GenServer.start_link(__MODULE__, start_from, opts)
  end

  def get(pid) do
    GenServer.call(pid, :get)
  end

  def init(start_from) do
    st = %{
      current: start_from,
      timer: :erlang.start_timer(@interval, self(), :tick)
    }

    {:ok, st}
  end

  def handle_call(:get, _from, st) do
    {:reply, st.current, st}
  end

  def handle_info({:timeout, _timer_ref, :tick}, st) do
    new_timer = :erlang.start_timer(@interval, self(), :tick)
    :erlang.cancel_timer(st.timer)

    {:noreply, %{st | current: st.current + 1, timer: new_timer}}
  end
end

This server increments a given number every 100ms and can report its state via OurNewApp.Counter.get/1:

iex -S mix
...
iex(1)> {:ok, pid} = OurNewApp.Counter.start_link(10000)
{:ok, #PID<0.182.0>}
iex(2)> OurNewApp.Counter.get(pid)
10136
iex(3)> OurNewApp.Counter.get(pid)
10146

Now let's integrate our server as a child. Update start/2 function in lib/our_new_app/application.ex to the following:

  def start(_type, _args) do
    children = [
      {OurNewApp.Counter, 10000}
    ]

    opts = [strategy: :one_for_one, name: OurNewApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

We see that our process starts automatically:

iex -S mix
...
iex(1)> [{_, pid, _, _}] = Supervisor.which_children(OurNewApp.Supervisor)
[{OurNewApp.Counter, #PID<0.141.0>, :worker, [OurNewApp.Counter]}]
iex(2)> OurNewApp.Counter.get(pid)
10119
iex(3)> Process.exit(pid, :shutdown)
true
iex(4)> Supervisor.which_children(OurNewApp.Supervisor)
[{OurNewApp.Counter, #PID<0.146.0>, :worker, [OurNewApp.Counter]}]

We queried the supervisor's children with Supervisor.which_children/1. We also see that our counter process restarted after we stopped it.

Our process tree now looks like this:

Adding GenServers to Custom Supervisors

Now let's make a special supervisor for our counter processes. Later, we'll see why we may want to do that. Our supervision tree will look like this:

First, we should make a callback module for our new special supervisor. Let's add lib/our_new_app/counter_sup.ex with the following content:

defmodule OurNewApp.CounterSup do
  use Supervisor

  def start_link(start_numbers) do
    Supervisor.start_link(__MODULE__, start_numbers, name: __MODULE__)
  end

  @impl true
  def init(start_numbers) do
    children =
      for start_number <- start_numbers do
        # We can't just use `{OurNewApp.Counter, start_number}`
        # because we need different ids for children

        Supervisor.child_spec({OurNewApp.Counter, start_number}, id: start_number)
      end

    Supervisor.init(children, strategy: :one_for_one)
  end
end

We must also update children for the main application supervisor in lib/our_new_app/application.ex:

  def start(_type, _args) do
    children = [
      {OurNewApp.CounterSup, [10000, 20000]}
    ]

    opts = [strategy: :one_for_one, name: OurNewApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

Let's see what we get:

iex -S mix
...
iex(1)> Supervisor.which_children(OurNewApp.Supervisor)
[{OurNewApp.CounterSup, #PID<0.161.0>, :supervisor, [OurNewApp.CounterSup]}]
iex(2)> Supervisor.which_children(OurNewApp.CounterSup)
[
  {20000, #PID<0.163.0>, :worker, [OurNewApp.Counter]},
  {10000, #PID<0.162.0>, :worker, [OurNewApp.Counter]}
]

That's just what we need: OurNewApp.Supervisor has OurNewApp.CounterSup as its child and OurNewApp.CounterSup has two OurNewApp.Counter children.

Many developers consider custom supervisors tricky and avoid using them. So let's do some simple exercises to get more acquainted with them.

First, we'll add a third counter to our counter supervisor at runtime:


iex(3)> new_child_spec = Supervisor.child_spec({OurNewApp.Counter, 30000}, id: 30000)
%{id: 30000, start: {OurNewApp.Counter, :start_link, [30000]}}
iex(4)> Supervisor.start_child(OurNewApp.CounterSup, new_child_spec)
{:ok, #PID<0.169.0>}
iex(5)> Supervisor.which_children(OurNewApp.CounterSup)
[
  {30000, #PID<0.169.0>, :worker, [OurNewApp.Counter]},
  {20000, #PID<0.163.0>, :worker, [OurNewApp.Counter]},
  {10000, #PID<0.162.0>, :worker, [OurNewApp.Counter]}
]

That was easy! With Supervisor.delete_child/2, Supervisor.restart_child/2, etc., we can easily manipulate the supervisor's children.

Secondly, instead of adding one worker to the existing tree, let's try adding a subtree with its own children (without a special module for the subtree supervisor):

iex(6)> children_specs = for n <- [10000, 20000, 30000], do: Supervisor.child_spec({OurNewApp.Counter, n}, id: n)
[
  %{id: 10000, start: {OurNewApp.Counter, :start_link, [10000]}},
  %{id: 20000, start: {OurNewApp.Counter, :start_link, [20000]}},
  %{id: 30000, start: {OurNewApp.Counter, :start_link, [30000]}}
]
iex(7)> hand_crafted_sup_spec = %{
...(7)>     id: :hand_crafted_sup,
...(7)>     start: {Supervisor, :start_link, [children_specs, [strategy: :one_for_one]]},
...(7)>     type: :supervisor,
...(7)>     restart: :permanent,
...(7)>     shutdown: 5000
...(7)> }
...
iex(8)> Supervisor.start_child(OurNewApp.Supervisor, hand_crafted_sup_spec)
{:ok, #PID<0.204.0>}
iex(9)> Supervisor.which_children(OurNewApp.Supervisor)
[
  {:hand_crafted_sup, #PID<0.204.0>, :supervisor, [Supervisor]},
  {OurNewApp.CounterSup, #PID<0.161.0>, :supervisor, [OurNewApp.CounterSup]}
]

The following took place:

hand_crafted_sup_spec was constructed, which started Supervisor.start_link
We told our main supervisor to start a child with this spec
The main supervisor started with children_specs parameters
It started counters from children_specs.

We could do this in another way: tell our main supervisor to launch an empty child supervisor, then add counters one by one to this child supervisor.

The process tree at the end of the experiment should look like this:

Examples of Custom Supervisor Usage

Let's see what happens if we terminate our app.

First, add some logging to lib/our_new_app/counter.ex:

  def terminate(reason, st) do
    Logger.info("terminating with #{inspect(reason)}, counter is #{st.current}")
  end

Also enable the :trap_exit flag for our counters, so that we can handle process termination — see terminate callback documentation:

  def init(start_from) do
    Process.flag(:trap_exit, true)

    st = %{
    ...
  end

Now, if we stop our application in the iex session, we see:

iex -S mix
...
iex(1)> Application.stop(:our_new_app)
19:35:43.544 [info]  terminating with :shutdown, counter is 20049
19:35:43.548 [info]  terminating with :shutdown, counter is 10050
19:35:43.548 [info]  Application our_new_app exited: :stopped
:ok

Imagine that we have to implement a graceful shutdown. The condition of gracefulness is to count up until we reach numbers divisible by 10 (10, 20, 30, etc) before shutdown.

Of course, in our simple example, we may just send ticks to count to the nearest number divisible by 10 in terminate.

Instead, imagine that these events are external end emulate some metrics that we would prefer to aggregate consistently.

First, let's add the possibility of a graceful restart to the OurNewApp.Counter module:

defmodule OurNewApp.Counter do
  use GenServer
  require Logger

  @interval 100

  def start_link(start_from) do
    GenServer.start_link(__MODULE__, start_from)
  end

  def get(pid) do
    GenServer.call(pid, :get)
  end

  def stop_gracefully(pid) do
    GenServer.call(pid, :stop_gracefully)
  end

  def init(start_from) do
    Process.flag(:trap_exit, true)

    st = %{
      current: start_from,
      timer: :erlang.start_timer(@interval, self(), :tick),
      terminator: nil
    }

    {:ok, st}
  end

  def handle_call(:get, _from, st) do
    {:reply, st.current, st}
  end

  def handle_call(:stop_gracefully, from, st) do
    if st.terminator do
      {:reply, :already_stopping, st}
    else
      {:noreply, %{st | terminator: from}}
    end
  end

  def handle_info({:timeout, _timer_ref, :tick}, st) do
    :erlang.cancel_timer(st.timer)

    new_current = st.current + 1

    if st.terminator && rem(new_current, 10) == 0 do
      # we are terminating
      GenServer.reply(st.terminator, :ok)
      {:stop, :normal, %{st | current: new_current, timer: nil}}
    else
      new_timer = :erlang.start_timer(@interval, self(), :tick)
      {:noreply, %{st | current: new_current, timer: new_timer}}
    end
  end

  def terminate(reason, st) do
    Logger.info("terminating with #{inspect(reason)}, counter is #{st.current}")
  end
end

Here we:

Add a terminator field to the state that keeps the address of the party that wants to stop the server
Set this field in the stop_gracefully handler
Continue counting until we get to a number divisible by 10
Respond to the terminating party and stop the server upon obtaining this number.

Let's see how that works for a single process:

iex -S mix
Erlang/OTP 23 [erts-11.0.4] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]

iex(1)> {:ok, pid} = OurNewApp.Counter.start_link(10000)
{:ok, #PID<0.167.0>}
iex(2)> OurNewApp.Counter.stop_gracefully(pid)
:ok
iex(3)>
20:03:13.061 [info]  terminating with :normal, counter is 10120

Everything works fine. But what stops all counters gracefully? As we see in the OTP docs, OurNewApp.Application.prep_stop is called (if it exists) before the application stops.

Let's add the desired functionality:

  @impl true
  def prep_stop(st) do
    stop_tasks =
      for {_, pid, _, _} <- Supervisor.which_children(OurNewApp.CounterSup) do
        Task.async(fn ->
          :ok = OurNewApp.Counter.stop_gracefully(pid)
        end)
      end

    Task.await_many(stop_tasks)

    st
  end

We also set :restart option to :transient in OurNewApp.CounterSup so that our counters do not restart after graceful shutdown:

Supervisor.child_spec({OurNewApp.Counter, {start_number, 200}},
  id: start_number,
  restart: :transient
)

Try to stop the app:

iex -S mix
iex(1)> Application.stop(:our_new_app)

20:24:02.958 [info]  terminating with :normal, counter is 10260

20:24:02.958 [info]  terminating with :normal, counter is 20260

20:24:02.962 [info]  Application our_new_app exited: :stopped
:ok

Now we only stop at numbers divisible by 10.

Using a special supervisor for our counters makes it possible to "find" all the instances quickly and operate with them. This is extremely important when applications go through start/stop stages.

Wrap-up

I hope you've managed to wrap your head around supervisors in Elixir, and have found this article helpful, alongside the previous article on hot code reloading.

There is another, even more complicated stage in an application life cycle: application code upgrades. We'll leave that for the third and final part of this series.

Until then, enjoy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Testing LiveView in Elixir

Roy Tomeij — 2021-08-17T00:00:00+00:00

In this two-part series, you'll get a comprehensive overview of everything you need to know to test your LiveView applications in Elixir.

In Part I, I'll introduce you to LiveView testing guidelines and you'll write some flexible and elegant LiveView unit tests. In Part II, you'll write interactive LiveView tests that validate a full set of live view behaviors.

Introducing LiveView's Powerful Testing Tools

If you've worked with LiveView, you've already experienced how productive you and your team can be with a framework that lets you build interactive UIs, while keeping your brain firmly focused on the server-side.

Testing LiveView is no different—you can exercise the full functionality of your live views with pure Elixir tests written in ExUnit through the help of the LiveViewTest module. So your tests are fast, concurrent, and continue to keep your focus firmly on server-side code.

LiveView's testing framework empowers you to quickly write robust and comprehensive tests that are highly stable. With a high degree of test coverage firmly in hand, you and your team will be able to move quickly and confidently when building LiveView applications.

We'll explore more on what makes LiveView testing so powerful, understand the principles for approaching live view tests, and write some unit tests.

In the next post, you'll write more advanced integration tests, and even test a distributed, real-time update feature in LiveView.

Let's get going!

Principles for Testing LiveView in Elixir

The basic principles for testing LiveView don't differ much from testing in other languages and frameworks. Any given test has three steps:

Set up preconditions,
Provide a stimulus, and
Compare an actual response to expectations.

We're going to write two kinds of tests—unit tests and integration tests—and we'll use this procedure for both types. Let's dig a little deeper into what these two kinds of tests look like in LiveView.

Unit Tests in LiveView

A unit test is designed to test an individual unit of code. Pure unit tests call one function at a time and then check expectations with one or more assertions.

Unit tests encourage depth. Such tests don't require much ceremony, so you can cover lots of scenarios quickly and easily. Unit tests also allow loose coupling because they don't rely on specific interactions between different parts of your code or your system.

We'll write some unit tests today to verify the behavior of the independent reducer functions that our live view implements to establish socket state under different conditions.

Integration Tests in LiveView

Integration tests, on the other hand, validate the interactions between different parts of your system. This kind of test offers testing breadth by exercising a wider swath of your application. The cost is tighter coupling since integration tests rely on specific interactions between parts of your system.

We'll write two integration tests that help us verify the behavior of the overall live view. Our first integration test will validate the interactions within a single live view process, and the second will verify a PubSub-backed interaction between two live views.

You might be surprised to hear that none of the tests we'll write in this post will be testing JavaScript. This is because the LiveView framework is specifically designed to handle JavaScript for us. For the average live view that doesn't implement any custom JavaScript, there's no need to test JS code or even to write any tests that use JavaScript. We can trust that the JS in the LiveView framework itself works as expected.

We're just about ready to start writing our very first LiveView test, and we'll start with some unit tests.

In general, it's a good idea to start with unit test coverage before moving on to integration testing. By exercising individual functions in unit tests with many different inputs, you can exhaustively cover corner cases. This lets you write a smaller number of integration tests to confirm that the complex interactions of the system work as you expect them to.

So, to recap, we can apply the following principles to LiveView testing:

Follow the three-step procedure of setting up preconditions, providing a stimulus, and validating expectations.
Write both unit and integration tests.
Don't test JavaScript you didn't write.
Start by writing comprehensive unit tests.

Alright, with these principles in hand, we're ready to start testing.

How to Unit Test LiveView

In this section, we'll use our three-step testing procedure to write a few unit tests for the behavior of a LiveView component.

We happen to be testing a LiveView component here, but the tests you'll learn how to write here apply to both components and regular live views.

As you build your tests, we'll look at how composing our LiveView component out of single-purpose reducer functions helps make the code easy to test.

The Feature

The testing examples that we'll be looking at in this post are drawn from the "Test Your Live Views" chapter in my book, Programming LiveView, co-authored with Bruce Tate. Check it out for an even deeper dive into LiveView testing and so much more.

In this example, we have an online game store that allows users to browse and review products. We provide our users with a survey that asks them for some basic demographic input, along with star ratings of each game. A LiveView component, SurveyResultsLive, displays this survey's results as a chart that graphs the games and their star ratings on a scale of 1 to 5. Here's a look at the component UI:

Notice the age group filter drop-down menu at the top of the chart. Users can select an age group by which to filter the survey results. When the page loads, the value of the age group filter should default to "all", and the chart should show all of the results. But when a user selects an age group from the menu, an event will be sent to the live view, and the results will be filtered.

Before we write our test, let's take a look at the code we're testing.

The Code

The update/2 function of the SurveyResultsLive component sets the initial state with the help of some single-purpose reducer functions. A reducer function is a function that takes in some input of some type and returns an updated version of that input.

We can use single-purpose reducer functions that take in a live view socket, add some state to that socket, and return an updated socket, to build nice clean pipelines for managing state in LiveView. Here's our update/2 function:

  def update(assigns, socket) do
    {:ok,
     socket
     |> assign(assigns)
     |> assign_age_group_filter()
     |> assign_products_with_average_ratings()
     |> assign_chart_svg()}
  end

The details of most of these reducer functions are not important, but we will take a closer look at the assign_age_group_filter reducer right now:

 def assign_age_group_filter(socket) do
  assign(socket, :age_group_filter, "all")
end

Our reducer is pretty simple. It adds a key :age_group_filter to the socket and sets it to the default value of "all".

However, we need a second version of this function to call in the handle_event/3 that gets invoked when the user selects an age group from the drop-down menu. This version of the function should take the selected age group from the event and set that as the value in the :age_group_filter key of socket assigns, like this:

def handle_event("age_group_filter", %{"age_group_filter" => age_group_filter}, socket) do
  {:noreply,
    socket
    |> assign_age_group_filter(age_group_filter)
    |> assign_products_with_average_ratings()
    #...
    }
end

def assign_age_group_filter(socket, age_group_filter) do
  assign(socket, :age_group_filter, age_group_filter)
end

Okay, with a basic understanding of this code in place, we're ready to write our test.

The Test

First off, we need to implement our test module in a file, gamestore/test/gamestor_web/live/survey_results_live_test.exs, like this:

defmodule GamestoreWeb.SurveyResultsLiveTest do
  use ExUnit.Case
  alias GamestoreWeb.SurveyResultsLive
end

Here, we're aliasing the SurveyResultsLive component to make it easier to refer to in our tests and we're using the ExUnit.Case behavior to gain access to ExUnit testing functionality. We're not importing any other code or even any other testing utilities since we can write our unit test in pure Elixir without any database interactions.

Next up, we'll establish a fixture that we can share across test cases. Every test will assert some expectations about a LiveView socket. So, we'll add a fixture that returns a socket, like this:

defmodule GamestoreWeb.SurveyResultsLiveTest do
  use ExUnit.Case
  alias GamestoreWeb.SurveyResultsLive

  defp create_socket(_) do
    %{socket: %Phoenix.LiveView.Socket{}}
  end
end

Now that our test module is defined and we've implemented a helper function to create test data (i.e. our socket), we're ready to write our very first test.

We'll start with a test that verifies the default socket state when no age group filter is supplied. Open up a describe block and add a call to the setup/1 function with a call to the helper function that returns a socket struct, like this:

defmodule GamestoreWeb.SurveyResultsLiveTest do
  use ExUnit.Case
  alias GamestoreWeb.SurveyResultsLive

  defp create_socket(_) do
    %{socket: %Phoenix.LiveView.Socket{}}
  end

  describe "Socket state" do
    setup do
      create_socket()
    end
  end
end

This ensures the socket will be available to all of our test cases. Now we're ready to write the test itself.

Create a test block within the describe block that pattern matches the socket out of the context we created in the setup function:

defmodule GamestoreWeb.SurveyResultsLiveTest do
  use ExUnit.Case
  alias GamestoreWeb.SurveyResultsLive

  defp create_socket(_) do
    %{socket: %Phoenix.LiveView.Socket{}}
  end

  describe "Socket state" do
    setup do
      create_socket()
    end

    test "when no age group filter is provided", %{socket: socket} do
    end
  end
end

Let's pause and think through what we're testing here and try to understand what behavior we expect to see. This test covers the function assign_age_group_filter/1. If it's working correctly, the socket should contain a key of :age_group_filter that points to a value of "all".

Now that we understand our expectation, let's finish up the test:

test "when no age group filter is provided", %{socket: socket} do
  socket =
    socket
    |> SurveyResultsLive.assign_age_group_filter()

  assert
    socket.assigns.age_group_filter == "all"
end

Great! We can see that our three-step testing process is represented here like this:

Set up preconditions by calling the setup function to establish our initial socket and make it available to the test case.
Provide the stimulus by calling the SurveyResultsLive.assign_age_group_filter/1 function to return a new socket.
Validate the expectations regarding the new socket's state.

Let's quickly add another test that validates the socket state when an age group filter is provided:

test "when age group filter is provided", %{socket: socket} do
  socket =
    socket
    |> SurveyResultsLive.assign_age_group_filter("18 and under")

  assert
    socket.assigns.age_group_filter == "18 and under"
end

Simple enough. Let's tackle a more complex scenario. Recall that our update/2 and handle_event/3 functions invoke assign_age_group_filter and then pipe the resulting socket into a call to the assign_products_with_average_ratings/1 function.

We won't worry about the details of this function for now. Just know that it queries for the product ratings given the filter info that is set in socket assigns. The composable nature of our single-purpose reducer functions lets us write multi-stage unit tests that exercise the behavior of the reducer pipeline as a whole.

Let's write a test that validates the product ratings assignment under different filter conditions.

Start by defining a new test and providing it with the socket from the setup function, like this:

test "ratings are filtered by age group", %{socket: socket} do

end

First, we'll set the socket state with a call to SurveyResultsLive.assign_age_group_filter/1 and validate that the products with average ratings are set correctly:

test "ratings are filtered by age group", %{socket: socket} do
  socket =
    socket
    |> SurveyResultsLive.assign_age_group_filter()
    |> SurveyResultsLive.assign_products_with_average_ratings()

  assert
    socket.assigns.products_with_average_ratings == all_products_with_ratings # don't worry about the value of `all_products_with_ratings`, just assume its a list of all products with their average ratings
end

Great! Next up, we'll update the socket's :age_group_filter state with the help of the assign_age_group_filter/2 reducer function, pipe the updated socket into another call to assign_products_with_average_ratings/1, and validate another assertion:

test "ratings are filtered by age group", %{socket: socket} do
  socket =
    socket
    |> SurveyResultsLive.assign_age_group_filter()
    |> SurveyResultsLive.assign_products_with_average_ratings()

  assert
    socket.assigns.products_with_average_ratings == all_products_with_ratings # don't worry about the value of `all_products_with_ratings`, just assume its a list of all products with their average ratings

  socket =
    socket
    |> SurveyResultsLive.assign_age_group_filter("18 and under")
    |> SurveyResultsLive.assign_products_with_average_ratings()

  assert
    socket.assigns.products_with_average_ratings == filtered_products_with_ratings # don't worry about the value of `filtered_products_with_ratings`, assume it's a list of all products with their average ratings provided by users in the specified age group
end

This test works, but it's pretty verbose. We can clean it up by taking inspiration from the clean, readable reducer pipelines we used in the SurveyResultsLive component to set state. Let's start by defining a helper function for use in our test, assert_keys:

defp assert_keys(socket, key, value) do
  assert socket.assigns[key] == value
  socket
end

This function takes in a socket, performs a test assertion, and then returns the socket. Since it takes in a socket and returns a socket, it behaves just like a reducer. Now, we can use our helper function to string together a beautiful, clean testing pipeline, like this:

test "ratings are filtered by age group", %{socket: socket} do
  socket
  |> SurveyResultsLive.assign_age_group_filter()
  |> SurveyResultsLive.assign_products_with_average_ratings()
  |> assert_keys(:products_with_average_ratings, all_products_with_ratings)
  |> SurveyResultsLive.assign_age_group_filter("18 and under")
  |> SurveyResultsLive.assign_products_with_average_ratings()
  |> assert_keys(:products_with_average_ratings, filtered_products_with_ratings)
end

Much better. Our usage of small, single-purpose reducer functions not only helped us write a clean, readable, and organized live view, it also helped us quickly write comprehensive unit tests.

Furthermore, we were able to string together a test pipeline that functions just like the real reducer pipeline in our live view. This let us write a unit test that exercised the pipeline's functionality as a whole.

You can easily imagine writing additional test cases for the other reducer functions implemented in the LiveView component, and additional pipeline tests that exercise the behavior of the reducer pipelines used in the update/2 and handle_event/3 functions of our component.

We applied our three-step testing process to end up with clean, readable test cases for both kinds of unit tests.

Next Up

In the next post, we'll leverage this same process to write some integration tests that verify the internal interactions within a single live view and the interactions between live view processes.

Until next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

A Guide to Hot Code Reloading in Elixir

Roy Tomeij — 2021-07-27T00:00:00+00:00

When building software, Elixir (or Erlang) offers great benefits, including concurrency, scalability and reliability.

In this series, we will examine how to make the most of these benefits in your production code upgrades. This article will focus on hot code reloading and upgrades. But before we dive in, let's quickly define OTP.

What Is OTP in Elixir?

Formally, Erlang/OTP is a specific implementation of Erlang Runtime System, i.e. a set of libraries, compilers and a VM implementation.

Informally, OTP often denotes a set of principles to build robust apps in Erlang and the corresponding set of built-in libraries.

Hot Code Reload: Tackling the Uncertainties

There is a bit of uncertainty about this concept.

When we speak of hot code reload or hot code upgrade, we usually mean an ability to change a running process behavior without any negative impact on that process. For example, we may change the behavior of a process that holds a TCP connection without terminating this connection.

Uncertainty comes in with scaling — the question is if we can upgrade:

a module
a package (an application in terms of OTP)
a whole running instance (a release in terms of OTP)

OTP offers tools for upgrading at any scale. In this article, we will consider application level upgrades.

How Are OTP and Hot Code Upgrades Related?

As we will see, hot code upgrades on a larger scale (application and release levels) work only for systems built according to OTP principles.

Hot Code Upgrades: The Basics

A good starting point to understand hot code reload is Hot Code Reloading in Elixir.

It explains the following key points:

How to reload code for a single module
How two versions of code exist after loading a new version of the module: new code and old code
The importance of external calls, which make it possible to transition from old code to new code
How GenServer helps us make such a transition seamlessly

At this point, I'd like to highlight one important concept in-depth: code purge.

Should You Code Purge in Elixir?

What happens if we want to upgrade code two or more times?

Let's create a small mix project:

mix new code_purge
cd code_purge

Then update lib/code_purge.ex to the following:

# lib/code_purge.ex
defmodule CodePurge do
  def pi do
    3.14
  end
end

Now we launch iex shell with mix:

iex -S mix
iex(1)> CodePurge.pi
3.14

Then update lib/code_purge.ex to:

# lib/code_purge.ex
defmodule CodePurge do
  def pi do
    3.142
  end
end

And recompile the project in a separate shell:

mix compile

In our iex shell, we reload the module code:

iex(2)> :code.load_file(CodePurge)
{:module, CodePurge}
iex(3)> CodePurge.pi
3.142

All has worked as expected. :code.load_file/1 found the updated Elixir.CodePurge.beam in _build/dev/lib/code_purge/ebin folder (as mix sets up code paths for us) and reloaded it.

But what happens if we try to reload this module once more, without actually changing it?:

iex(4)> :code.load_file(CodePurge)
{:error, :not_purged}

What Went Wrong Here?

Wow, that doesn't work. This is because Erlang can't have two versions of old code.

To overcome this, there are two other methods of :code: :code.purge/1 and :code.soft_purge/1.

A purge evicts the old code:

iex(5)> :code.purge(CodePurge)
false
iex(6)> :code.load_file(CodePurge)
{:module, CodePurge}

We can upgrade the code of the module again after the purge. But why do we even need to control that? Why not purge code automatically?

Well, there may still be processes running old code, and we should decide what to do with them during the upgrade. This is also why there are two functions:

:code.purge/1 — kills processes running old code
:code.soft_purge/1 — fails if there are any processes running old code

This leads to important consequences: if we want to upgrade our code more than once, our processes will be killed by default during upgrades.

Let's illustrate this.

How Not to Do a Code Upgrade

First, add file lib/code_purge/pi.ex to your toy project with the following content:

# lib/code_purge/pi.ex
defmodule CodePurge.Pi do
  def start_link do
    spawn_link(&server/0)
  end

  def server do
    receive do
      {:get, from} ->
        send(from, {:ok, 3.14})
        CodePurge.Pi.server()
    end
  end

  def get(pid) do
    send(pid, {:get, self()})

    receive do
      {:ok, value} ->
        {:ok, value}
    after
      1000 ->
        :error
    end
  end
end

Then, run iex shell, spawn a server and check everything is fine:

iex(1)> pid = CodePurge.Pi.start_link()
#PID<0.140.0>
iex(2)> CodePurge.Pi.get(pid)
{:ok, 3.14}

Now, reload the module once (without any actual changes to functions) and try to purge it so that you can do the next 'upgrade':

iex(3)> :code.load_file(CodePurge.Pi)
{:module, CodePurge.Pi}
iex(4)> :code.purge(CodePurge.Pi)
** (EXIT from #PID<0.152.0>) shell process exited with reason: killed

What Happened Here?

As expected, your server just died, and even an external call to CodePurge.Pi.server/0 couldn't save you. The server didn't receive messages and so didn't transition to the new code after the first upgrade.

This isn't robust. One of the obvious reasons for the failure is that we didn't use OTP libraries (GenServer and related libraries) dedicated to creating this kind of server.

Avoid Spawn in Real-World Software Development

In many books and articles, we see code examples demonstrating the power of Elixir or Erlang: tons of processes easily spawned directly with spawn or spawn_link.

However, in real-world software development, we generally should avoid creating home-brewed servers or other long-running processes, and should instead use OTP libraries.

Even for 'one-off' asynchronous tasks, we shouldn't directly use spawn or spawn_link.

Elixir has a great alternative for spawn, though: Task module (covered in depth in the AppSignal article Demystifying processes in Elixir).

How To Do a Code Upgrade Using GenServer

Let's create a better version of our server in lib/code_purge/pi_gs.ex:

# lib/code_purge/pi_gs.ex
defmodule CodePurge.PiGs do
  use GenServer

  def start_link(value \\ 3.14) do
    GenServer.start_link(__MODULE__, value)
  end

  def init(value) do
    {:ok, value}
  end

  def handle_call(:get, _from, value) do
    {:reply, value, value}
  end

  def get(pid) do
    GenServer.call(pid, :get)
  end
end

And now, try to upgrade/purge the code of a running process several times:

iex(1)> {:ok, pid} = CodePurge.PiGs.start_link()
{:ok, #PID<0.161.0>}
iex(2)> CodePurge.PiGs.get(pid)
3.14
iex(3)> :code.load_file(CodePurge.PiGs)
{:module, CodePurge.PiGs}
iex(4)> :code.purge(CodePurge.PiGs)
false
iex(5)> :code.load_file(CodePurge.PiGs)
{:module, CodePurge.PiGs}
iex(6)> :code.purge(CodePurge.PiGs)
false
iex(7)> CodePurge.PiGs.get(pid)
3.14

Nothing bad happens! The reason why is easy to understand.

Our pid process doesn't spin in CodePurge.PiGs code. It runs a GenServer loop, and we don't update the GenServer module code at all.

CodePurge.PiGs is a callback module, and the name is kept in a GenServer internal state. GenServer makes external calls to CodePurge.PiGs functions when serving GenServer requests.

The main challenge is to keep updating the states of GenServer processes, so that any new code can work.

For a single GenServer, this can be done through :sys module and code_change callback of GenServer. This is covered in depth in the previously mentioned hot code reloading article, here, we'll only briefly demonstrate it.

Without closing the previous iex session, let's update lib/code_purge/pi_gs.ex to the following and compile:

# lib/code_purge/pi_gs.ex
defmodule CodePurge.PiGs do
  use GenServer

  def start_link(value \\ 3.14) do
    GenServer.start_link(__MODULE__, value)
  end

  def init(value) do
    {:ok, [value]}
  end

  def handle_call(:get, _from, st) do
    [value] = st
    {:reply, value, st}
  end

  def get(pid) do
    GenServer.call(pid, :get)
  end

  def code_change(_old_vsn, value, _extra) do
    {:ok, [value]}
  end
end

In code_change we updated the state, just wrapping it with a list. We also updated handle_call and init callbacks. Now, in the existing iex session, run:

iex(8)> :code.purge(CodePurge.PiGs)
false
iex(9)> :sys.suspend(pid)
:ok
iex(10)> :code.load_file(CodePurge.PiGs)
{:module, CodePurge.PiGs}
iex(11)> :sys.change_code(pid, CodePurge.PiGs, nil, [])
:ok
iex(12)> :sys.resume(pid)
:ok
iex(13)> CodePurge.PiGs.get(pid)
3.14
iex(14)> :sys.get_state(pid)
[3.14]

Everything works fine, and the last call to :sys.get_state demonstrates that the state has actually changed.

Wrap-up

In the first part of this series, we've seen that a GenServer implementation is needed for effective hot code upgrades. We've also demonstrated how to upgrade a single GenServer instance consistently.

Upgrading an individual process, together with its callback module, can be used as a 'tactical weapon' to fix localized bugs or add some logging.

But updating a system at a greater scale, on a regular basis, requires more powerful tools. In the next part of the series, I'll delve into the world of supervisors in Elixir.

I hope you found this run-through of hot code reloading useful. See you next time for supervisors!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Building Aggregates in Elixir and PostgreSQL

Roy Tomeij — 2021-07-13T00:00:00+00:00

In this article, I will describe the roles and implementation of aggregates in Elixir and PostgreSQL. We'll learn:

what domain invariants are
how to use aggregates to protect them
what to pay attention to when choosing a specific implementation for your aggregates
why domain knowledge is essential when designing suitable aggregates

We have a lot of ground to cover, so let's dive right in!

What Are Domain Invariants?

Let’s start by looking at a core concept: domain invariants. A domain invariant is a fact that always has to be true in your business domain. Here are some examples:

In an e-commerce platform, a cart's total has to be the sum of all product prices and applied discounts.
In a booking application, a room cannot be booked by two different people at the same time.
In accounting software, the current account balance has to be the sum of all the transactions and a withdrawal cannot be made when the balance is below 0.
With a carpooling app, there can be a max of X people in the car (where X is the maximum capacity of the car).

Aggregates Protect Invariants

Now let's look at aggregates. The job of an aggregate is to protect the invariants for a given entity or group of entities.

Let's continue with the example of the carpooling application. One of our invariants is that the number of people who sign up for a ride should never exceed the number of seats:

defmodule Ride do
  def book(ride_id, passenger_id, now \\ DateTime.utc_now()) do
    ride = fetch_ride(ride_id)

    if can_add?(ride, passenger_id) do
      passenger = insert_passenger(ride_id, passenger_id)
      {:ok, ride, passenger}
    else
      {:error, :cannot_add}
    end

  end

  defp can_add?(ride, passenger_id) do
    # Any conditions we need to check
    limit_not_exceeded?(ride) && passenger_not_blocked?(ride, passenger)
  end
end

If we don't explicitly control concurrency, nothing prevents two different processes from booking the same ride when there's only one seat left:

Depending on how we model the data and the updates, we might end up with different results.

If we insert a participant as a new row, we might end up with more participants.

If we keep the list of participants as a list on the Ride, the participant added by Process B might be overridden by the one added by Process A (which is known as a Lost Update Problem).

There is often more than one invariant that we need to protect. Examples from our carpooling app might include:

If one user blocks another one, they should not be able to ride in the same car.
A premium user can choose the front seat if there's no other premium user already booked.
For multi-stop rides, you cannot add a new stop if it conflicts with the existing route (according to some business rules).

Editing any of the information related to a ride can endanger any of those invariants. That's why we need to have a proper boundary to run all the checks and validations.

The Aggregate and the Root

An aggregate needs to have a required boundary of consistency. Within that boundary, all invariants have to always be true. Every potential update we make to the aggregate should include all the necessary checks.

Typically, we use the aggregate root to help us do that. The aggregate root is the "main" entity of our aggregate.

In our example, the Ride is the aggregate root, while the entire aggregate consists of the Ride and all Participants associated with that ride.

The general rule is that all updates for any entities in the aggregate should go through the aggregate root and validate the invariants across the entire aggregate.

Every update done to one of the aggregate's entities has the potential of breaking one of the invariants. A single path for all the updates allows us to verify all of the changes in a consistent way.

How to Implement Aggregates in Elixir

Our goal is to make sure that all the invariants are true after saving an aggregate.

Since we essentially want to perform proper concurrency control, we need to do the following for every change:

Make sure no one else can change the resource until we're finished.
Check all the invariants and rules.
Make all the necessary changes.
Allow others to use the resource.

Let's look at some ways of achieving that.

The Process-based Approach

One of the jobs of an aggregate is to prevent conflicting, concurrent updates. In Elixir, we have a tool for that — a process.

A process handles incoming messages sequentially. If we can ensure that a single process does all operations on an aggregate, conflicting updates should not be a problem.

We'll need a way to find a process for each aggregate. Registry is a nice tool for that:

defmodule Ride do
  use GenServer

  def start_link([]) do
    GenServer.start_link(__MODULE__, [], name: get_name(ride_id))
  end

  def add_passenger(ride_id, passenger_id) do
    GenServer.call(get_name(ride_id), {:add_passenger, passenger_id})
  end

  def handle_call({:add_passenger, passenger_id}, _from, state) do
    # The calls are processed sequentially

    if can_add?(state, passenger_id) do
      new_state = insert_passenger(state.ride_id, passenger_id)
      {:reply, ok, new_state}
    else
      {:reply, {:error, :no_seats_left}, state}
    end
  end

  defp get_name(ride_id) do
    {:via, Registry, {RideRegistry, {__MODULE__, ride_id}}}
  end
end

The biggest issue with the process-based approach is that problems start appearing when we run our code on more than one node.

Registry only runs locally. If two nodes attempt to do operations on the same entity, nothing prevents them from doing so simultaneously.

Solutions like the global registry won't really help, either, since they generally favor availability over consistency. This means that, in the event of node partition (which will happen, sooner or later), two nodes can end up registering their own processes for a specific entity, leading to the same problem as the local registry.

The chances of that happening are lower than with Registry, but it's still not a safe solution.

The Database Approach

We can use a database as a source of truth for our needs. If your database supports locks, you can use that mechanism to prevent concurrent updates.

The necessary steps for updates are:

Fetch the resource you want to modify using the FOR UPDATE lock.
Verify all the invariants.
Make updates.
Release the lock.

Firstly, we need to lock the resource that we're trying to modify. Since there can be another process in the middle of another transaction, we need to make sure that we fetch the latest version of the resource — hence the FOR UPDATE option for the lock.

This guarantees that if there's another process which has already locked that resource, our process will wait for its turn and fetch the most recent, already committed version of that resource:

defmodule Ride do
  def add_passenger(ride_id, passenger, now \\ DateTime.utc_now()) do
    Repo.transaction(fn ->
      ride =
        Ride
        |> lock("FOR UPDATE")
        |> where(id: ^ride_id)
        |> Repo.one()

      if can_add?(ride, passenger_id) do
        passenger = insert_passenger(ride_id, passenger_id)
        {:ok, ride, passenger}
      else
        {:error, :cannot_add}
      end
    end)
  end

  defp can_add?(ride, passenger_id) do
    # Any conditions we need to check
    limit_not_exceeded?(ride) && passenger_not_blocked?(ride, passenger)
  end
end

Since the locks are automatically released once a transaction is committed, we don't need to do that explicitly:

This approach is called pessimistic locking. As the name implies, we want to make sure that nobody else is doing another operation even before we start.

There are some problems with this approach.

First, no matter what the operation is, we have to wait until a lock is released before doing any other work. This can be a problem if the steps necessary to perform the business transaction are long.

If transactions last a long time, and only one can happen at a time, we risk lowering the throughput, increasing the latency, and hurting the user experience.

There's another problem with waiting. Since the lock has to happen inside a database transaction, the process must hold to that transaction for the entire operation. This is a problem because of the limited pool size of database connections. Every process waiting to acquire a lock occupies a connection that could be used for some other query.

Finally, this approach won't easily work for business transactions that span multiple HTTP requests. If the business transaction is more complicated and requires some input from the user, pessimistic locking is probably not a good choice.

We can address these problems by using optimistic locking instead.

Optimistic Version Control

In optimistic locking, you rely on the assumption that no other process will change the aggregate.

Whether that's a safe assumption to make depends on the specific use case. For example, the chances of a conflict are different for an e-commerce system, where a user only operates on their own cart, versus a booking application — where multiple people might want to book a concert ticket at the same time.

Understanding your domain and specific use case is important!

The steps for optimistic locking are as follows:

You start a transaction and fetch the resource without locking it.
You verify all the constraints and do all the work without actually committing the changes.
Just before you commit, you lock the resource and verify if the version hasn't changed (which would mean that some other process overwrote the resource) and bump the version.
If the previous step passes, you commit the transaction. If not, you roll back the changes and return an error.

def add_passenger(ride_id, passenger_id) do
  ride = fetch_ride(ride_id)

  Repo.transaction(fn ->
    with true <- can_add?(ride, passenger_id),
      :ok <- insert_passenger(ride, passenger_id),
      :ok <- check_and_update_version(ride, ride.version + 1) do
      :ok
    else
      {:error, :versions_dont_match} -> Repo.rollback(:version_conflict)

      # Other error handling skipped
    end
  end)
end

defp fetch_ride(ride_id) do
  Repo.get(Ride, ride_id) |> Repo.preload(:passengers)
end

def check_and_update_version(ride, new_version) do
  query =
    Ride
    |> where(id: ^ride_id)
    |> where(version: ^ride.version)

  case Repo.update_all(query, set: [version: new_version]) do
    {1, _} -> :ok
    # No record found, so the version must have changed in the meantime
    {0, _} -> {:error, :versions_dont_match}
  end
end

Notice that we used a single update query here to verify the invariant. This works because Postgres prevents two or more transactions from making conflicting updates by double-checking the where clauses. See the 'Read Committed Isolation Level' section from the docs for more information.

With optimistic locking, most of the work is done before we lock the resource.

In general, there's no need to use a transaction for the changes. If you can tolerate the state being inconsistent for a bit (which might also be a necessity in a distributed environment), instead of rolling back the transaction, you can use any other mechanism (like the saga pattern) to roll back the changes manually after detecting a conflict.

There's also nothing preventing you from making the business transaction span across multiple requests, giving you more flexibility (the only thing you need to do is pass the initial version id from request to request or save it somewhere).

Optimistic Locking — Alternatives to Version

The exact solution you use depends on the specific use case and the business logic behind it. While keeping the version number is a generic solution that works for every case (and is supported by Ecto!), you can make it better by using domain expertise.

In many cases, detecting a version conflict does not mean that we have to roll back all the changes. In our example, we might keep a seats_left field on each ride. Instead of verifying the version number (and potentially returning an error for transactions that are correct from a business standpoint), we might use that field for concurrency control. As long as the number is greater than 0, we allow the update:

defp decrease_seats_left(ride_id) do
  query =
    Ride
    |> where(id: ^ride_id)
    |> where([r], r.seats_left > 0) # Protects our invariant

  case Repo.update_all(query, inc: [seats_left: -1]) do
    {1, _} -> :ok
    {0, _} -> {:error, :no_seats_left}
  end
end

This helps both the user experience and the performance of the system (since fewer queries are made).

Domain Knowledge is Key

Concurrency control might seem like a purely technical problem, but it's often not. Carefully analyzing the domain can lead to a far superior (and simpler) solution than relying on generic technical patterns. It's useful to:

Carefully analyze the domain invariants.
Make the invariants explicit in the codebase.
Understand the chances of invariant violations.
Understand the consequences of invariant violations.

You'll realize that the problem is not as black-and-white as we often think. Looking at it from the technical perspective, we see a strictly correct or incorrect state.

There will be certain cases where fully protecting all the invariants will be too hard or even harmful in real life. Plane overbooking is one example where the invariant is violated (to some degree) on purpose.

In these instances, designing processes to detect and deal with invariant violations might be better than trying to prevent those violations.

Events, CQRS, and Read and Write Models

When talking about aggregates, it's really easy to go down the DDD rabbit hole. You often see aggregates emitting events, or using separate data models for reads and writes. It's easy to start thinking about event sourcing.

Are those techniques useful? Of course!

The thing is that you don't have to solve all of them at once. It's also really hard to do most of the cases.

Designing well-isolated, event-sourced aggregates might seem easy on paper or when starting from scratch.

Most of the time, though, you'll be working on a messy codebase that you want to make better (which is fine!).

In such a scenario, building a fully event-driven aggregate is really challenging. Making sure that the events can cover all of the use cases and possible states is a lot of work.

But we can still benefit a lot just from making the aggregates a bit more explicit. Remember: the aggregate's job is not to emit events or use a write model, it's to protect domain invariants.

One of the best first steps you can do when refactoring an existing codebase is to find those invariants, put them in the code, and validate them in an explicit way.

Summary

In this article, we've learned that:

A domain invariant always has to be true in a particular domain.
Concurrent updates can lead to compromised invariants and Lost Update problems.
Elixir processes can be used for concurrency control, but it's hard to do so with more than one node.
Locking a resource before using it is called 'pessimistic concurrency control'.
'Optimistic concurrency control' is when we assume that it's safe to do an operation but still verify this just before committing changes.

Maybe most importantly, we've seen that domain knowledge is essential and that it's crucial to take small steps in the right direction instead of trying to do everything right the first time.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How to Test WebSocket Clients in Elixir with a Mock Server

Roy Tomeij — 2021-06-22T00:00:00+00:00

In this post, we'll give a very high-level overview of how to implement a long-running connection between two services with WebSocket, and then we'll write unit tests for functionality.

The WebSocket protocol allows two-way communication between two channels. Most developers know it as a fast, near real-time communication medium between a client and a server.

In the Phoenix and Elixir world, many recognize Websockets from abstractions built on top of it, like LiveView or Channels.

But WebSocket is also a great communication medium for an app built on microservices architecture. Let's say we have an e-commerce application that uses (among others), two services:

Orders — Create and manage the user's orders
Payments — Process the payments

When a user makes a purchase, the Orders service creates an order corresponding to the purchase and sends it to the Payments service. This service can then respond with a payment page URL that the user needs to visit to complete his payment. After completion, the Orders service needs to create an invoice for the order and send it to the user, so it will need a way to watch for the completion of payments.

For such an architecture, a long-running connection between Orders and Payments through a WebSocket is a great choice, as both services can interchange messages and notify the other service about updates or events.

Setting Up a WebSocket Client

WebSockex is a library for implementing a WebSocket client in Elixir. Here's a very simplified implementation of how the client could look on the Orders service.

The client behaves as follows:

It is started as a process with a URL of the Payments WebSocket Server and some details about the order it needs to process.
As soon as it's connected, it sends a message to the Payments service to initiate the payment process.
At some point (normally soon after the payment has been initialized), the Payments service responds with a list of available payment methods for the order.
The client selects the first payment method and asks Payments to initiate payment with that method.
Payments sends a message with the payment page url that the Orders service can use to initiate a payment on the web app.
At some point, Payments sends a message to the client informing them about the state of the transaction, e.g. FULFILL, CANCEL etc.
If the payment state is FULFILL, then fulfill the order. If the payment state is CANCEL, then cancel the order and close the connection.

Testing External Services

Now that we have a working WebSocket client in place, the next question is: how do we test it? It connects to an external service (this service is ultimately controlled by us, but is still external in terms of the Orders service).

There are several ways to write tests that interact with external services:

Create a separate service for the API Client and stub out those requests with Mox from José Valim.
Mock out the requests during the tests with a library like Mock.
Roll out your own web server to handle the requests.

All these methods and their pros and cons have been covered in detail in several posts in the community. One that I particularly like and recommend is Testing External Web Requests in Elixir? Roll Your Own Mock Server.

We'll discuss how to write tests for the above client by rolling out our own server during the tests.

Creating a Mock Server

First, we need to create a mock server that will provide a connection to, and interact with the client.

We will use plug and cowboy to create the server and control the socket.

Set Up a Mock HTTP Server

Since we might need several of these servers running in parallel during the tests, we'll need a way to generate ports so that the servers don't collide.

We can do this by starting an Agent that begins randomly at a port and then assigns us a port incrementally during the tests:

defp get_port do
  unless Process.whereis(__MODULE__), do: start_ports_agent()

  Agent.get_and_update(__MODULE__, fn port -> {port, port + 1} end)
end

defp start_ports_agent do
  Agent.start(fn -> Enum.random(50_000..63_000) end, name: __MODULE__)
end

The mock server now needs to obtain a port and start listening on that port for connections.

On each new client connection, we will open a socket (more on this later). This is the server-side socket that is linked with the client-side socket opened from our WebSocket client implementation.

This socket is running in its own process, so we'll need to obtain its PID inside the tests for our test code to be able to interact with it.

Since the tests only know about the HTTP server and not the socket, we'll send the socket PID back to the server so we can retrieve it from the test code. This is how the mock server code looks (I know it's a bit complex, but don't be alarmed — it'll be easy to understand once we see the socket).

Using the above server is very simple:

Start the server:

{:ok, {server_ref, url}} = Commerce.Orders.MockWebsocketServer.start(self())
on_exit(fn -> Commerce.Orders.MockWebsocketServer.shutdown(server_ref) end)

Connect the client, using the above url.
If you need to send messages to the client, first obtain the server pid:
```
server_pid = Commerce.Orders.MockWebsocketServer.receive_socket_pid()
```
Send messages to the server that will be relayed to the client. Check Commerce.Orders.TestSocket.websocket_info/2 for all possible clauses:
```
send(server_pid, {:send, {:text, frame}})
```
This server also sends all the messages that it receives to the owner process. This means that to verify that the server has received a message, you can use:
```
assert_receive on the owner:
assert_receive("SOME MESSAGE")
```

Set Up the Mock Socket

After creating the mock server, you then create the actual server-side socket that will handle the connections. This is the Commerce.Orders.TestSocket our mock HTTP server dispatches the initial connection to.

The full code for the test socket can be found here. Let's break it down.

The init is what is called after MockWebsocketServer starts and it receives a new connection request from our WebSocket client (that we want to test). You can check out the full cowboy_websocket documentation for more details on the supported responses. [{test_pid, agent_pid}] is the initial state that we sent out from the MockWebsocketServer.dispatch and we will keep track of that in state to manage sending frames to the socket.

websocket_init is called once the connection is upgraded to WebSocket. This is where we send the server socket's pid to the MockWebsocketServer, which the test code can then retrieve using receive_socket_pid.

websocket_handle is called for every frame received. This is where we further process the received frame to verify messages from our client. If this differs between different WebSocket clients that we want to test, you could skip the call to handle_websocket_message and its implementations — just use send(state.pid, to_string(msg)) to send all messages to the test process. The test process can then verify the frames as required.

websocket_info is called for every Erlang message that the process receives. This is what we use to receive messages from the test code, to trigger certain events from the server socket.

The first argument to this function could be anything that the test code sends, so you can add as many clauses as you want to support to simulate certain events.

Here, we are only interested in initiating a close from the test code or sending a particular frame to the client socket.

Test Code

Now that we have the WebSocket client, mock server and mock socket out of the way, we can get to the actual test code that will test the WebSocket client using the mocks.

To start the server in our tests, we will use MockWebsocketServer.start:

setup do
  {:ok, {server_ref, url}} = MockWebsocketServer.start(self())
  on_exit(fn -> MockWebsocketServer.shutdown(server_ref) end)

  %{url: url, server_ref: server_ref}
end

Let's test out our client with the mock server in place.

Instead of writing several smaller tests that test a single part of the client, I am providing a big test case that describes the full interaction. This is so I can describe all available testing options without too much boilerplate code:

test "interacts with the server", %{url: url, order: order} do
  {:ok, pid} = WebSocketClient.start_link(%{url: url})
  server_pid = MockWebsocketServer.receive_socket_pid()

  # A payment is initiated immediately after a connection
  assert_receive Jason.encode!(%{initiate_payment: true})

  # WebSocket client receives the payment methods from the server (from the hard coded message response). We do not need any code for this since we generalized it directly in the socket.

  # The payment is initiated with credit card
  assert_receive Jason.encode!(%{initiate_payment: "credit_card"})

  # Send a FULFILL state from the server. This is the alternative to hard coded messages in the server side socket.
  send(server_pid, {:send, {:text, Jason.encode!(%{state: "FULFILL"})}})

  # Confirm that the order was fulfilled
  assert {:ok, %Order{state: "FULFILL"}} = Orders.get_order(order.id)
end

While it takes some work to set up our mock server and socket to test our WebSocket clients, it makes our testing really smooth and simple. Later, if we add another WebSocket client, we can utilize the same server to unit test the new client without any complex setup or mocking.

In addition to this, we now have tests that do not depend on any of the client's implementation details. We are just testing business logic rather than implementation or tying our test code to specific libraries being used in the implementation.

So if you were to replace the WebSocket client implementation to use gun instead of WebSockex, we wouldn't need to change anything in our tests. In fact, our tests would serve as an additional validation point for this migration, to ensure that everything keeps working as it did before.

Wrap-up

In the post, we briefly touched on a sample microservices architecture, looked at how we could use WebSockets to communicate between different services and saw a very simplified implementation of a WebSocket client using WebSockex.

If you are looking for a more complex WebSockex client example, this post walks through building a complete STOMP client with WebSocket.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

How Absinthe Uses Compilation Callbacks for Schema Validation in Elixir

Roy Tomeij — 2021-01-19T00:00:00+00:00

Absinthe manages to do a lot of interesting things during its compilation process, and today we're going to look a bit at how that works. We'll look closely at how it uses some metaprograming tricks and module attributes to provide compile-time schema validation for us.

It's pretty amazing (to me, at least) that when we use Absinthe, we can have a really simple, easy-to-use API to define our schema and we still get a good amount of compile-time type checking out of it! For example, if we try and use a type that hasn't yet been defined, we'll see an error like this in our terminal when we try and compile our application:

== Compilation error in file lib/blog_web/schema.ex ==
** (Absinthe.Schema.Error) Invalid schema:
/home/devon/sandbox/absinthe_tutorial/lib/blog_web/schema/account_types.ex:10: User_state :custom_enum is not defined in your schema.

  Types must exist if referenced.


    (absinthe 1.4.16) lib/absinthe/schema.ex:271: Absinthe.Schema.__after_compile__/2
    (stdlib 3.13.2) lists.erl:1267: :lists.foldl/3
    (stdlib 3.13.2) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir 1.11.2) lib/kernel/parallel_compiler.ex:314: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

The fact that this happens is cool on its own, but how they manage to do this is what I think is really cool. It takes a lot of really tricky (but interesting) usage of modules and module attributes to make it work, and that's what we'll be covering today. But before we can get to the actual type checking, we need to take a quick look at how one defines a schema with Absinthe, and then how that schema is compiled to create those modules and module attributes using Elixir compilation callbacks.

Defining a Schema with Absinthe

To define our GraphQL schema using Absinthe, we need to write a single module in which that schema is declared, and in that module we need to use Absinthe.Schema. If your schema is small enough then doing that in one file is easy enough:

defmodule BlogWeb.Schema do
  use Absinthe.Schema

  alias BlogWeb.Resolvers

  object :user do
    field :id, :id
    field :name, :string
    field :posts, list_of(:post) do
      resolve &Resolvers.Content.list_posts/3
    end
  end

  object :post do
    field :id, non_null(:id)
    field :title, non_null(:string)
    field :body, non_null(:string)
    field :user, non_null(:user)
  end

  input_object :post_params do
    field :id, non_null(:id)
    field :title, non_null(:string)
    field :body, non_null(:string)
    field :user_id, non_null(:id)
  end

  query do
    field :posts, list_of(:post) do
      resolve(&Resolvers.Content.list_posts/3)
    end
  end

  mutation do
    field :create_post, :post do
      arg(:params, non_null(:post_params))
      resolve(&Resolvers.Content.create_post/3)
    end

    field :update_post, :post do
      arg(:params, non_null(:post_params))
      resolve(&Resolvers.Content.update_post/3)
    end

    field :delete_post, :post do
      arg(:id, non_null(:id))
      resolve(&Resolvers.Content.delete_post/3)
    end
  end
end

However, once you start building out your application and things get bigger, you generally end up breaking the schema up into multiple "schema fragment" files and importing the types defined in those fragments into your schema using the Absinthe.Schema.Notation.import_types/2 and the Absinthe.Schema.Notation.import_fields/2 macros.

To do that with our schema above we might end up doing something like what is below, with each set of types defined in its own module, each of which calls use Absinthe.Schema.Notation. We can imagine that each module is defined in its own file, although they technically don't need to be:

defmodule BlogWeb.Schema.UserTypes do
  use Absinthe.Schema.Notation

  alias BlogWeb.Resolvers

  object :user do
    field :id, :id
    field :name, :string
    field :posts, list_of(:post) do
      resolve &Resolvers.Content.list_posts/3
    end
  end
end

defmodule BlogWeb.Schema.PostTypes do
  use Absinthe.Schema.Notation

  alias BlogWeb.Resolvers

  object :post do
    field :id, non_null(:id)
    field :title, non_null(:string)
    field :body, non_null(:string)
    field :user, non_null(:user)
  end

  input_object :post_params do
    field :id, non_null(:id)
    field :title, non_null(:string)
    field :body, non_null(:string)
    field :user_id, non_null(:id)
  end

  object :post_queries do
    field :posts, list_of(:post) do
      resolve(&Resolvers.Content.list_posts/3)
    end
  end

  object :post_mutations do
    field :create_post, :post do
      arg(:params, non_null(:post_params))
      resolve(&Resolvers.Content.create_post/3)
    end

    field :update_post, :post do
      arg(:params, non_null(:post_params))
      resolve(&Resolvers.Content.update_post/3)
    end

    field :delete_post, :post do
      arg(:id, non_null(:id))
      resolve(&Resolvers.Content.delete_post/3)
    end
  end
end

defmodule BlogWeb.Schema do
  use Absinthe.Schema

  import_types(Absinthe.Type.Custom)
  import_types(BlogWeb.Schema.UserTypes)
  import_types(BlogWeb.Schema.PostTypes)

  alias BlogWeb.Resolvers

  query do
    import_fields(:post_queries)
  end

  mutation do
    import_fields(:post_mutations)
  end
end

But how does Absinthe know that when we're referencing the :post type in the definition of our :user type, the :post is a valid type to use? Well, that's where the fun stuff come in!

How Elixir's Compilation Callbacks Work

Well, to know how Absinthe works its magic, first we need to know a bit about Elixir's compilation callbacks. A compilation callback is, as it sounds, a function that is executed either before, during, or after compilation takes place. There are a three compilation callbacks, but the two we care about for today are the @before_compile and @after_compile callbacks.

These are two functions that are called, as you would assume, before and after compilation of a module. The before_compile callback receives as an argument the compilation __ENV__, which is a struct containing information about the compilation process. More info on what exactly is in there can be found in the docs for Macro.Env. Likewise, the after_compile callback receives that same compilation __ENV__, and also the compiled bytecode for the module.

These two callbacks give us the opportunity to set up some things that might be needed for compilation in our before_compile callback, and then some checking of things that have just been compiled in our after_compile callback. That's exactly how Absinthe uses those two features for its schema compilation and schema validation.

How Absinthe Does Schema Validation at Compile Time

So, what exactly is Absinthe doing when it compiles? Well, let's start with the compilation of those schema fragments. Absinthe.Schema.Notation contains a definition of a __before_compile__/1 function which is used as the handler for the @before_compile callback for each of those schema fragments.

defmacro __before_compile__(env) do
  module_attribute_descs =
    env.module
    |> Module.get_attribute(:absinthe_desc)
    |> Map.new()

  attrs =
    env.module
    |> Module.get_attribute(:absinthe_blueprint)
    |> List.insert_at(0, :close)
    |> reverse_with_descs(module_attribute_descs)

  imports =
    (Module.get_attribute(env.module, :__absinthe_type_imports__) || [])
    |> Enum.uniq()
    |> Enum.map(fn
      module when is_atom(module) -> {module, []}
      other -> other
    end)

  schema_def = %Schema.SchemaDefinition{
    imports: imports,
    module: env.module,
    __reference__: %{
      location: %{file: env.file, line: 0}
    }
  }

  blueprint =
    attrs
    |> List.insert_at(1, schema_def)
    |> Absinthe.Blueprint.Schema.build()

  [schema] = blueprint.schema_definitions

  {schema, functions} = lift_functions(schema, env.module)

  sdl_definitions =
    (Module.get_attribute(env.module, :__absinthe_sdl_definitions__) || [])
    |> List.flatten()
    |> Enum.map(fn definition ->
      Absinthe.Blueprint.prewalk(definition, fn
        %{module: _} = node ->
          %{node | module: env.module}

        node ->
          node
      end)
    end)

  {sdl_directive_definitions, sdl_type_definitions} =
    Enum.split_with(sdl_definitions, fn
      %Absinthe.Blueprint.Schema.DirectiveDefinition{} ->
        true

      _ ->
        false
    end)

  schema =
    schema
    |> Map.update!(:type_definitions, &(sdl_type_definitions ++ &1))
    |> Map.update!(:directive_definitions, &(sdl_directive_definitions ++ &1))

  blueprint = %{blueprint | schema_definitions: [schema]}

  quote do
    unquote(__MODULE__).noop(@desc)

    def __absinthe_blueprint__ do
      unquote(Macro.escape(blueprint, unquote: true))
    end

    unquote_splicing(functions)
  end
end

At first the code in that function might be tricky to understand, but the most important part of understanding what's going on there is looking at the definition of the __absinthe_blueprint__/0 function. We can see that we're defining a function that returns a map, and that map contains a lot of information about the state of things before the current schema fragment was compiled. This __absinthe_blueprint__/0 function will be really important in the final compilation step that we'll look at in a bit.

One other really intersting thing about this code this is important to notice is how many calls to Module.get_attribute/2 there are! This is one of the things that Absinthe leans on heavily for this compilation process - the use of modules and module attributes as essentially defining global variables that can be accessed by other modules during their compilation! There are a lot of calls to Module.get_attribute/2 and Module.put_attribute/3 in this module, and recognizing this pattern helps us put the rest of the process into context.

The other thing happening here is that we're defining a lot of functions in a dynamically named module! These functions contain yet more information, and we can see a bit more of how this is used in the __before_compile__/1 function defined in Absinthe.Schema:

defmacro __before_compile__(_) do
  quote do
    @doc false
    def __absinthe_pipeline_modifiers__ do
      [@schema_provider] ++ @pipeline_modifier
    end

    def __absinthe_schema_provider__ do
      @schema_provider
    end

    def __absinthe_type__(name) do
      @schema_provider.__absinthe_type__(__MODULE__, name)
    end

    def __absinthe_directive__(name) do
      @schema_provider.__absinthe_directive__(__MODULE__, name)
    end

    def __absinthe_types__() do
      @schema_provider.__absinthe_types__(__MODULE__)
    end

    def __absinthe_types__(group) do
      @schema_provider.__absinthe_types__(__MODULE__, group)
    end

    def __absinthe_directives__() do
      @schema_provider.__absinthe_directives__(__MODULE__)
    end

    def __absinthe_interface_implementors__() do
      @schema_provider.__absinthe_interface_implementors__(__MODULE__)
    end

    def __absinthe_prototype_schema__() do
      @prototype_schema
    end
  end
end

When each schema fragment is defined, it also defines a module that contains the information about the module that was just defined - so for example, for our BlogWeb.Schema.UserTypes module that we used above, it will define a BlogWeb.Schema.UserTypes.Compiled module. With this convention, it allows Absinthe know where to look for information for each module that was compiled with some schema information.

And now that all that work has been done during the compilation process, we can look at the __after_compile__/2 callback defined in Absinthe.Schema:

def __after_compile__(env, _) do
  prototype_schema =
    env.module
    |> Module.get_attribute(:prototype_schema)

  pipeline =
    env.module
    |> Absinthe.Pipeline.for_schema(prototype_schema: prototype_schema)
    |> apply_modifiers(env.module)

  env.module.__absinthe_blueprint__
  |> Absinthe.Pipeline.run(pipeline)
  |> case do
    {:ok, _, _} ->
      []

    {:error, errors, _} ->
      raise Absinthe.Schema.Error, phase_errors: List.wrap(errors)
  end
end

This is where all that information and all that metaprogramming is actually used for some helpful user features! In short, that callback will use all of the information that's been stored in various module attributes and exposed by defining all of those different functions in all of those .Compiled modules to build up something that Absinthe calls a blueprint. This blueprint is again what it sounds like - it contains the information for how documents will later by evaluated against the current GraphQL schema during resolution. It then evaluates this blueprint, and if there are any errors returned from that evaluation they're raised at the end of the compilation process!

Clearly this is kind of a compilcated process, but it's also a cool way to use some of the basic features of the Elixir compiler to deliver value to users. Exploring this process helped me learn a lot about this method of compilation of applications, but it also made it clear to me that the Absinthe team has put a great deal of time and effort into making this user experience really great, and for that I'm very thankful!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

In your terminal, create a new Elixir app like so:

mix new blog_app --sup

Then open up mix.exs and add the following dependencies to enable us to work with Ecto:

# mix.exs

defmodule BlogApp.MixProject do
  use Mix.Project

  ...

  defp deps do
    [
      {:ecto_sql, "~> 3.2"},
      {:postgrex, "~> 0.15"}
    ]
  end
end

Setting Up the Repo

One thing to note is that Ecto has adaptors for PostgreSQL, MySQL and SQLserver by default. Here, we are adding the PostgreSQL adaptor (we assume you already have PostgreSQL installed). Next, run mix deps.get to install the newly added dependencies. Then we need to create a repo which forms the point of contact between the app and the database:

mix ecto.gen.repo -r Blog.Repo

Go ahead and edit the automatically created database configuration:

# config/config.exs

import Config

config :blog_app, Blog.Repo,
  database: "blog_app_repo",
  username: "user",
  password: "password",
  hostname: "localhost"

To ensure the Ecto process is started when the app starts, we need to add the repo to the app's supervisor like so:

# lib/blog_app/application.ex

defmodule BlogApp.Application do

  use Application

  @impl true
  def start(_type, _args) do
    children = [
      Blog.Repo
    ]

    ...

  end
end

Next, we add the Blog repo to config.exs:

# config/config.exs

import Config

...

config :blog_app, ecto_repos: [Blog.Repo]

And finally, run mix ecto.create to create the database and finalize the setup process. Next, let's do a migration for a Post model that we'll use in the proceeding steps.

Adding Ecto Migrations

Run the commands below to create the Post and associated Comment migration:

mix ecto.gen.migration create_posts -r Blog.Repo

mix ecto.gen.migration create_comments -r Blog.Repo

This will generate two time-stamped migrations in priv/repo/migrations which you should edit as shown below, starting with the Post:

# priv/repo/migrations/XXXXXX_create_posts.exs

defmodule Blog.Repo.Migrations.CreatePosts do
  use Ecto.Migration

  def change do
    create table(:posts) do
      add(:title, :string)
      add(:body, :string)

      timestamps()
    end
  end
end

Also edit the Comment migration as follows:

# priv/repo/migrations/XXXXXX_create_comments.exs

defmodule Blog.Repo.Migrations.CreateComments do
  use Ecto.Migration

  def change do
    create table(:comments) do
      add(:body, :string)
      add(:post_id, references(:posts), null: false)

      timestamps()
    end
  end
end

With that done, run the migrations with mix ecto.migrate. Finally, to use our app we need a structured way to query the database using Ecto schemas.

Adding Ecto schemas

Although it's possible to use schema-less queries, let's avoid having to manually construct queries every time we need to use them.

We'll add the first schema for posts. Create a new file under lib/blog/post.ex with the following contents:

# lib/blog/post.ex

defmodule Blog.Post do
  use Ecto.Schema

  schema "posts" do
      has_many :comments, Blog.Comment # Here we add the asscociation to comments
    field :title, :string
    field :body, :string

    timestamps()
  end

end

We won't go into the details of how to build schemas for now. If you need to dig into the topic further, check out the Ecto schema docs.

Next, create another schema to handle comments under lib/blog/comment.ex and edit it as below:

# lib/blog/comment.ex

defmodule Blog.Comment do
  use Ecto.Schema

  schema "comments" do
      belongs_to :post, Blog.Post  # Here we add the asscociation to post
    field :body, :string

    timestamps()
  end

end

And with that, the app is ready. Let's now use it to dive straight into learning more on Ecto associations.

Direct Casting of Association ID's: `cast`

Working with associations doesn't always have to be complex. In a situation where you have the target ID, Ecto lets you treat the relation column as a normal database field.

To give a concrete example, let's assume we work with two models, Post and Comment, where multiple comments can refer to a single post. In that case, your models would look something like this.

    defmodule Blog.Post do
        use Ecto.Schema
        schema "posts" do
            has_many :comments, Blog.Comment
            field :title, :string
            field :body, :string
        end
    end

    defmodule Blog.Comment do
        use Ecto.Schema
        schema "posts" do
            belongs_to :post, Blog.Post
            field :body, :string
        end
    end

These models reflect the following table schemas in the database:

Each table contains a primary field id by default. The has_many field on Post does not refer to a database field, it only exists to hint to Ecto that it's possible to preload comments for a post using the comment's belongs_to field. The belongs_to field, on the other hand, refers to an existing field in a table schema. By default, the name of this field in a table is different from the name in Ecto's model: the database field has _id at the end.

Ecto lets you modify these kinds of association fields the same way you would modify any other field. In Ecto, changing the value of a primitive field is called "casting". If you need to create a new comment for a particular post, you don't really need any of the association-specific functions, you can just cast the value of a primary key:

comment
    |> cast(params, [:post_id, :body])

This is the simplest and most straightforward method of creating an association between two tables.

Casting Associations: `cast_assoc`

It is useful to think about cast_assoc as a special version of cast that works on associations. However, casting associations can be much more complex than casting normal fields. The cast call normally translates more or less directly into a single SQL query, while cast_assoc might result in multiple INSERT, UPDATE or DELETE queries. Let's assume the database tables from the previous example contains the following content:

Post:

id	title	body
7	A story...	Once upon a time...

Comment

id	post_id	body
10	7	Great story!
11	7	What happened next?
12	7	Thanks for the article

Since the Post model contains has_many association to Comment, it's trivial to preload all comments on a particular post:

Post
  |> Repo.get!(id)
  |> Repo.preload(:comments)

The shape of the returned data will be as follows:

%Post{
    "id" => 1,
    "title" => "A story of...",
    "body" => "Once upon a time...",
    "comments" => [
        %Comment{"id" => 10, "body" => "Great story!"},
        %Comment{"id" => 11, "body" => "What happened next?"},
        %Comment{"id" => 12, "body" => "Thanks for the article"},
    ],
}

Single cast_assoc call on :comments will replace the association as a whole. In effect, this means that the values you pass to cast_assoc will be returned in future preload calls. This does not necessarily mean that all database rows are replaced. Ecto compares before and after states and does the minimal amount of work required to reach the desired state. To illustrate that, consider the following changeset:

params = %{comments: [
    %Comment{"id" => 11, "body" => "What happened next?"},
    %Comment{"id" => 12, "body" => "Thank you for the post"},
    %Comment{"body" => "Interesting"},
]}
post
|> cast(params, [])
|> cast_assoc(:comments)

Executing this changeset results in three calls to the database:

DELETE the comment with an id of 10
UPDATE the comment with the id of 12 and set body to "Thank you for the post"
INSERT a comment with a body "Interesting" and assign it a new id.

The row with the ID of 11 was left unchanged because it matches preloaded values. An important thing to note here is that Ecto will not preload data on its own, so to make use of cast_assoc, you need to remember to call preload beforehand. However, you are not restricted to preloading a complete association. cast_assoc will work just as well when you use preload as a subset of records with Repo.preload(:comments, query). This feature is very useful for limiting the impact of cast_assoc to a subset of associated records.

Defining Associations: `put_assoc`

At first glance, put_assoc is in many ways similar to cast_assoc: it also works on a whole association and requires you to pre-load records to be updated. However, upon closer examination, it turns out to be almost opposite in the way you use it. The crucial distinction is that put_assoc is designed to update the association "references", not the data. That is, you would typically use put_assoc when you want to connect a record to one or more records that already exist in the database.

put_assoc can be used to associate a new comment with an existing post, similar to what we did in the "direct casting" section, but without using the post_id field directly:

post = Repo.get!(7)
# ...
comment
    |> cast(params, [:body])
    |> put_assoc(:post, post)

This makes your code a little bit cleaner in cases with complex primary fields because Ecto does all the bookkeeping for you.

Building Related Records: `build_assoc`

build_assoc is the convenience function that allows you to create related records through an association on an existing record. To continue our post/comment example, here is another way to create a new comment:

post = Repo.get!(7)
...
comment_params = %{
    "title": "A story..."
    "body": "Once upon a time..."
}
Ecto.build_assoc(post, :comments, comment_params)
# %Comment{post_id: 7, title: "A story...", body: "Once upon a time..."}

The power of build_assoc is in its expressiveness: the code above clearly shows you that comment belongs to the post.

Unlike the functions discussed above, build_assoc does not operate on a changeset — it builds one. This means you would only ever use build_assoc when you want to create a new record.

Wrap Up

While this post doesn't begin to cover the variety of use-cases you might encounter in production applications, I hope it gives you a strong foundation from which to begin searching for an answer. And if you need a refresher in the future, here is a simple flowchart that will remind you of the discussed use cases:

Ecto's association functions are relatively thin abstractions over field references in a database. Understanding how each of those functions works on a database level is crucial to becoming an expert Elixir/Phoenix developer. Fortunately, Ecto is built in such a way that each function is relatively small, deterministic, and has a single purpose. After you have mastered the basics, you can expect fewer "gotchas" compared to the traditional ORMs (or at least in my experience that was the case). Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Capabilities of Elixir's Logger

Roy Tomeij — 2020-10-13T00:00:00+00:00

This post was updated on 9 August 2023 to reference the latest version of the Elixir logger.

Logs are an important part of your application and logging shouldn't be one of the last things you think of. You should configure your log system, formatter, and style as soon as you start the development of your app. Also, do your best to document the process and share how it works with the rest of your team.

In this article, we're going to demonstrate how logs work in Elixir. We'll jump into Elixir's Logger module, which brings a lot of power to logging features. It also smoothly integrates with Erlang’s :logger to convert code to Elixir syntax.

You’ll learn how to customize your formatter, set up and change the log defaults of a Phoenix web API, as well as understand Logger levels a bit better.

Logs are useful but having a complete overview of your app is even better. Check out our 5-in-1 application monitoring for Elixir. We're free for OSS.

Our Project Environment Setup

To follow along, you should have a working Elixir environment set up. You can find instructions for that on Elixir’s Installation Page.

After installation, the following command will be available:

iex

When you run that command, an Elixir Interactive Shell will be opened. This is a REPL command-line environment in which you can test Elixir commands.

This is the perfect stage for some initial tests with Elixir’s IO module. As the name suggests, this module handles operations over IO, like data input and output.

To check that out, run the following commands within the iex shell:

IO.puts("Hello world from Elixir")

IO.puts(:stdio, "Logging from the stdio...")
IO.puts(:stderr, "This is an error...")

IO.inspect(self())

This is the simplest form of logging in Elixir. The puts function takes two arguments, the first being the target destination of the log message (as an atom) and the second, the message itself.

When called with only the message argument, IO.puts will print to :stdout by default.

The third line exemplifies how you could change the destination to something else, like the :stderr. Although this setting changes the internal structure of the log, it doesn’t change anything in the output because there’s no color highlight being made.

Finally, there’s an inspect function on the last line. This function inspects the more complex structures of your Elixir application and allows you to log, for example, information about the process id in which that code is running.

You can see the logs of this execution below:

Phoenix Project

For the purposes of the article, you'll also need to have a Phoenix application created from scratch.

So make sure to follow the official instructions here.

Since our focus is on logging, we only need the homepage and a single GET endpoint.

What About the Elixir Logger?

Among other things, Elixir’s Logger module provides:

All the major levels of logging common to most languages
Pre-formatting of log messages before they reach their destination
Integration with Erlang’s Logger for even more power

Logger Levels

Logger currently embraces the following log levels:

:debug: must be used for debug-only purposes. Be careful not to set this mode for production environments.
:info: the most used one. It should be triggered whenever you need to print information of any nature.
:warn and :error: even though :info can be used for everything, it’s better to finetune your warning and error messages. This way, automation tools will understand your categorized logs better.

They are stacked in the exact order of precedence which Elixir considers when having to choose between one or another.

Usage Example

Its usage is pretty simple. See the following example with a string interpolation:

Logger.info("Hey, it's me: #{user_name}")

You can also use it within the iex shell. Just run the following:

require Logger
user_name = "Julio"
Logger.info("Hey, it's me: #{user_name}")

Here's the output:

11:29:55.379 [info]  Hey, it's me: Julio

Formatter

Note that, by default, Logger comes with a preset format. In the above log, you can see a timestamp of when the event happened, the level of the log, and its message.

When you work with real-world applications, like a web API, you need to define which levels are accepted for each environment.

Get back to the Phoenix application you created previously and open the config folder. You may see a bunch of Elixir files with names that match development, testing, and production environments. Open the config.exs file, which is the boss of your app configuration and locate the config :logger line.

Take a brief look at these lines. This is Elixir’s Logger configuration. Whatever you change here, is going to be applied to all your app’s logs.

Start by changing the default log level of the entire app. If you specify that it should be error (the highest in the stack), then the others won't be logged anymore. So change the first line of the config to this:

config :logger, :console,
    level: :error

To test, you’ll need to add some logs to your controller function. So, open the file simple_controller.ex that’s located in the web/controllers folder and switch its code to this:

require Logger

defmodule SimpleElixirApi.PageController do
  use SimpleElixirApi.Web, :controller

  def index(conn, _params) do
    IO.puts("I'm a log")
    Logger.debug("I'm a debug log")
    Logger.info("I'm an info log")
    Logger.warn("I'm a warn log")
    Logger.error("I'm an error log")

    Logger.metadata(request_id: "123")
    render(conn, "index.html")
  end
end

Restart your server and hit the http://localhost:4000/ address again. You should see the following in your console:

Change the level to the others and check out the results. Remember that you may configure each of the environment files separately to avoid log setting confusion.

Message Formatter

Alternatively, you can format your log messages by providing a regex. Change your Logger config (at configs.ex) to the following:

config :logger, :console,
    level: :info,
    format: "$time $message $metadata[$level] \n"

Note that the level was changed to allow more log lines for comparison.

Here, the format is receiving only the log’s variables to be injected into the output. However, you can customize with any of Elixir’s regex that may please you.

This should be the output:

In the end, it hasn’t changed that much. However, this was on purpose. For more advanced logs, you’ll need to write your own formatter.

Creating Your Formatter

Elixir’s config allows you to create and set your own formatter. With it, you have the option of creating whatever routines you want before, during, and after each log reaches its destination.

To make this happen, you’ll need to create a new logger module. So, within the lib folder, create another one called logger and create a file named my_logger.ex inside it with the following code:

defmodule Logger.MyLogger do
    def format(level, message, timestamp, metadata) do
      "time=#{timestamp} level=[#{level}] metadata=#{inspect(metadata)} message=#{message}\n"
    rescue
      _ -> "Ooops, there was an error here!\n"
    end
end

You can name the function whatever you want but it’s common practice to give it the format name. It basically receives all common log attributes and formats them into a predefined pattern.

The message obeys the usual pattern for maps (key=value), very common among log processing tools.

In case anything goes wrong, an error message will be thrown at the rescue block.

For it to run, the Elixir config file has to be aware of it, so change the logger config to the following:

config :logger, :console,
    level: :info,
    format: {Logger.MyLogger, :format}

Restart the server and check out how your logs look now.

Logging Metadata

Another useful feature that comes built-in with Elixir’s Logger is metadata logging.

If you're using AppSignal to monitor your app, instead of adding metadata to your logs, you can add sample data to your AppSignal samples by using tagging. This way you can supply extra context on errors and performance issues.

There are times when you need to group a bunch of information in categorized logs. Since you’re under Phoenix's wing, your web APIs may need to automatically show some data related to HTTP requests and responses, like the request’s id.

Logger metadata works as a list of keywords (similar to maps and dictionaries) that’s exclusive to the running process (which means that, in the HTTP fandom, one request won’t get affected by others).

Every time a request arrives in the house, Logger will catch it, check your log configs and ask what metadata keywords are allowed to be logged.

You can define them one by one or tell Logger to consider them all:

config :logger, :console,
    level: :info,
    format: {Logger.MyLogger, :format},
    metadata: [:request_id]

To set them all, just change the metadata config to:

metadata: :all

Do you remember the code line you added to the controller a few moments ago:

Logger.metadata(request_id: "123")

Now you can check out the request’s id appearing in your logs:

12:08:13.040 metadata=[request_id=123] [info] My request log

Logging to Files

Perhaps one of the easiest features within Logger is changing the log’s destination to a file rather than the default console.

First, you need to set up a new backend for the logger_file_backend package:

config :logger,
    backends: [{LoggerFileBackend, :debug_log}]

Then, you specify the destination file that Logger will send the logs to:

config :logger, :debug_log,
    path: 'myLog.log',
    level: :debug

When you tail -f your myLog.log file, you should see the logs.

Conclusion

It’s unbelievable how such a simple subject as logging can lead to a lot of unseen extra information. Right now, Elixir's Logger is tightly integrated with Erlang’s, so that developers can now implement custom loggers. Good luck with exploring Elixir loggging!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Monitoring Any System with StatsD and AppSignal's Standalone Agent

Roy Tomeij — 2020-09-23T00:00:00+00:00

Monitoring your application alone is not always enough to get the full picture of your systems. Frequently, services running in satellite apps (or supporting apps) can have an acute impact on your day-to-day operations. Database servers are well-known examples of this. Backup scripts and other background jobs can also slow systems and are often overlooked.

The AppSignal APM for Node.js, and the Ruby APM and Elixir APM automatically instrument your app itself. But AppSignal does not watch these satellite processes by default. To extend monitoring everywhere and have all your data in a single app, you can install AppSignal’s standalone agent.

AppSignal's Standalone Agent

The standalone agent is based on the same software with which we usually instrument Ruby, Elixir, or JavaScript applications. This software can also run in standalone mode.

The standalone agent can be used to monitor:

Infrastructure: machines that are part of our system but do not run application code.
Background jobs: such as intensive cron jobs or long-running data-processing scripts. If these background jobs are written in a supported language (Ruby, Elixir, or Node.js) you can use the standard integration.
More languages: programs written in languages other than those supported out of the box.

For instance, with the standalone agent, we could track a machine learning model written in Python, instrument backup scripts, monitor Kafka brokers, or collect host metrics in our web farms. You can view all this information on AppSignal to complement the metrics you already have for your main applications.

How It Works

The agent is shipped as a deb or rpm package and doesn’t have any language dependencies. It runs in any Debian/Ubuntu or Red Hat-based systems. For detailed installation instructions, check the agent documentation.

Once installed, the agent is configured in a few minutes and keeps running forever as a daemon, silently monitoring your infrastructure. What is more, the agent includes a StatsD server that relays any custom data you log into your AppSignal dashboard.

What Is StatsD?

StatsD is a standard for collecting and aggregating arbitrary data. It focuses on logging metric and performance information. It uses a lightweight text protocol over UDP connections that have a small footprint in your machine.

A StatsD message looks like this:

KEY:VALUE|TYPE

Where KEY is any arbitrary string, and VALUE is a number. The type value defines how the number is processed.

We support three types of metrics:

c: this is a counter that increments each time it is called. For instance, active_users:1|c adds 1 to the active_users counter.
g: a gauge takes a numeric value and maintains it until updated. This is useful to record values that change up and down over time, such as throughput, number of active users, or number of pending tasks in a queue.
t: stores timing values. This type is ideal for tracking durations. AppSignal calculates means, count, and percentiles for all the logged timings.

Other unsupported metric types will be silently ignored.

Sending Data to StatsD

The standalone agent listens for UDP packets on port 8125. We can send StatsD-formatted strings from the command line using netcat:

echo -n "myscript.myevent.counter:1|c" | nc -4u -w0 localhost 8125

Since we’re using UDP, we don’t have to wait for a response.

Socat also works:

echo -n "myscript.myevent.counter:1|c" | socat - udp:localhost:8125

This makes it easy to instrument any batch or cron jobs. For instance, the following lines use a gauge to log how much data a backup job has generated:

backup_size=$(du -m /backups | cut -f1)
echo -n "backup.data:$backup_size|g" | nc -4u -w0 localhost 8125

We’re not limited to integers. StatsD also works with floating-point numbers:

echo -n "network.latency:0.2|g" | nc -4u -w0 localhost 8125

Using Tags

You can add tags to your metrics. The StatsD server supports optional tags at the end of the message:

KEY:VALUE|TYPE|#TAGS

We can apply several tags in the same message and assign values for filtering later:

echo -n "backup.data:$backup_size|g|#backups,env:production" | nc -4u -w0 localhost 8125

We’ll learn how to see the data in AppSignal in a bit.

Instrumenting Languages

The StatsD server is compatible with any language that can send UDP packets. Let’s see a few examples.

Python is a popular language for data mining and machine learning. We can add instrumentation into Python applications using the build-in socket library:

import time

# measure time taken for function
start = time.process_time()
train_my_model()
training_time = time.process_time() - start

# send value to statsd
import socket

UDP_IP = "127.0.0.1"
UDP_PORT = 8125
MESSAGE = b"model.training.time:" + bytes(str(training_time), 'utf-8') + b"|t"

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))

We could achieve similar results in Java with the java.net library:

byte[] buffer = "mybatch.exception.counter:1|c".getBytes();
InetAddress address = InetAddress.getByName("127.0.0.1");
DatagramPacket packet = new DatagramPacket(
    buffer, buffer.length, address, 8125
);
DatagramSocket datagramSocket = new DatagramSocket();
datagramSocket.send(packet);

PHP is another example of an incredibly popular language. We can send UDP packets with socket_sendto:

$msg = "mywebsite.active_users:$ACTIVE_USERS|g";
$len = strlen($msg);

$sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_sendto($sock, $msg, $len, 0, '127.0.0.1', 8125);
socket_close($sock);

StatsD Clients

Thus far, we used the built-in networking capabilities in every language. But there’s more. Many languages and products include third-party StatsD clients or addons. Any StatsD-compliant client should work, at least for the supported data types. You can find a list of clients at the StatsD project wiki.

Viewing Host Data in AppSignal

The Host Metrics dashboard shows your machine’s resource utilization:

AppSignal adds an entry for every machine running the standalone agent.

The dashboard shows the load and CPU averages disk, network, memory, and swap usage:

Creating Dashboards

AppSignal doesn’t automatically generate dashboards for the StatsD value you sent—you’ll need to create custom dashboards for this.

First, click on Add dashboard under the dashboard menu:

Give a name to the dashboard:

Clicking the Add graph button shows the graph builder:

Give a name to the graph, and click on Add metric:

Pick the metric you’re interested in from the menu.

Optionally, use tags for filtering:

Next, click Back to overview. You might want to try different graph types and value units to find out which one best suits the data you want to represent.

Timing data looks better with area graphs because the mean and percentiles are shown more clearly.

Line graphs work great for counters and gauges.

Once you’re happy with the result, click on Create graph.

Note that you can also add a dashbaord for any other automatically instrumented or measured metrics if you have AppSignal set of as your performance monitoring tool (APM) for Node.js, or your Ruby (on Rails) APM or Elixir APM.

Wrapping Up

We’ve learned how to use AppSignal’s standalone server to watch your machines and satellite code. With its built-in StatsD server, you can record arbitrary performance data and instrument any process.

Check the following links to learn more about using the standalone agent:

Testing the Tricky Parts of an Absinthe Application

Roy Tomeij — 2020-08-19T00:00:00+00:00

Today, we hope to make testing Absinthe a bit easier for you. We believe that it's a great library for writing GraphQL applications, but if you previously haven't done much work on an Absinthe application, you might find some things a bit tricky to test.

The worst part of this is that some of these really tricky things to test are some of the best parts of Absinthe, and so, with them being a bit hard to test, folks might end up not using those parts of the library as much as they should.

Today's Ingredients

The main ingredient today is a GraphQL schema representing a blog. There are also some references to things that we're not going to show, but to avoid unnecessary complexity, we will assume that they're there and working as expected. For example, we're not going to be looking at the "application" logic in modules like MyApp.Comments, as we don't need to do that in order to understand the testing that we'll carry out.

Here's our main ingredient: the GraphQL schema below that represents a blog.

defmodule MyAppWeb.Schema do
  use Absinthe.Schema

  import Absinthe.Resolution.Helpers

  alias MyApp.{Comments, Posts, Repo, User, Users}

  @impl true
  def context(ctx) do
    loader =
      Dataloader.new()
      |> Dataloader.add_source(Comments, Dataloader.Ecto.new(Repo))
      |> Dataloader.add_source(Posts, Dataloader.Ecto.new(Repo))
      |> Dataloader.add_source(Users, Dataloader.Ecto.new(Repo))

    Map.put(ctx, :loader, loader)
  end

  # This is only public so we can show how to test it later 😀
  def resolve_unread_posts(user, _, %{loader: loader}) do
    loader
    |> Dataloader.load(Users, :posts, user)
    |> Absinthe.Resolution.Helpers.on_load(fn loader ->
      unread_posts =
        loader
        |> Dataloader.get(Users, :posts, user)
        |> Enum.filter(& !&1.is_read)

      {:ok, unread_posts}
    end)
  end

  object :user do
    field(:name, non_null(:string))
    field(:age, non_null(:integer))
    field(:posts, non_null(list_of(non_null(:post))), resolve: dataloader(Posts))
    field(:unread_posts, non_null(list_of(non_null(:post)), resolve: &resolve_unread_posts/3)
  end

  object :post do
    field(:title, non_null(:string))
    field(:body, non_null(:string))
    field(:is_read, non_null(:boolean))
    field(:comments, non_null(list_of(non_null(:comment))), resolve: dataloader(Comments))
  end

  object :comment do
    field(:body, non_null(:string))
    field(:user, non_null(:user), resolve: dataloader(Users))
  end

  query do
    field(:users, non_null(list_of(non_null(:user))), resolve: fn _, _, _ -> Repo.all(User) end)
  end
end

What to Test and Where

When you've got a GraphQL API and you're using Absinthe, it means you generally have three "layers" in your application that you can test. They are (from the outermost layer to the innermost layer):

Document resolution — where you actually send a GraphQL document as a user would and resolve that document
Resolver functions — which are just functions and so can be tested in the normal way that you'd test any other function
Your application functions — which is basically everything else 😀

Like in any other application, you'll be writing tests at each of these levels. The number of tests you write at each level, and what you test, is often a matter of personal preference.

Reason to Test at These Levels in Absinthe

The important thing is: because of how certain kinds of behavior are separated in Absinthe, and in the way it resolves documents, there is some behavior that can only be tested at some levels. For example, if you're using the default resolution function for a field in an object, you can only test the resolution of that field at the document resolution level.

Similarly, if you're using any of the Absinthe.Resolution.Helpers.dataloader functions, you won't be able to test that behavior anywhere but at the document resolution level. This is a bit of a pattern actually — using Dataloader is basically a necessity for most GraphQL applications, but using it also makes that behavior a bit harder to test and also forces us to test certain behavior at a higher level, in a more expensive test than one might want.

Testing Document Resolution

So let's focus on testing at document resolution. Since we know we're going to have to write some tests where we're resolving an actual document, we should ensure that those tests are as valuable to us as they can be! Since these will already be rather expensive tests given that they cover the entire stack, you might as well try and squeeze all the value out of them that you can. These tests will end up being rather large, and hopefully, you won't have to have too many of them.

One thing I see somewhat frequently is folks trying to make these tests smaller and more manageable, but this comes with some potential issues. One of the great things about GraphQL is that clients can send a document that only requests the data they need, making it easy to compose bits of functionality together into a larger API.

However, this means that it's also really easy to accidentally miss functionality when testing an object's resolution! This can lead to errors when resolving a field in a type that isn't seen until that field is actually requested by a client in production, and that's not good.

So, the thing that I've relied on is the rule that when I'm testing document resolution, I always request every field in whichever object I'm testing.

How to Request Every Field in a Test

But how do we make this easy to do? Luckily, there's a function for that! In the assertions library, there are some helpers for testing Absinthe applications. Included in those helpers, is the document_for/4 function which automatically creates a document with all fields in the given object and will also recursively include all fields in any associated objects to a given level of depth! The default there is 3, but since we want 4 levels deep we'll need to override that. So, instead of a test that looks like this:

test "resolves correctly", %{user: user} do
  query = """
  query {
    users {
      name
      age
      posts {
        title
        body
        isRead
        comments {
          body
          user {
            name
            age
          }
        }
      }
      unreadPosts {
        title
        body
        isRead
        comments {
          body
          user {
            name
            age
          }
        }
      }
    }
  }
  """

  assert {:ok, %{data: data}} =
           Absinthe.run(query, MyAppWeb.Schema, context: %{current_user: user})

  assert %{
            "users" => [
              %{
                "name" => "username",
                "age" => 35,
                "posts" => [
                  %{
                    "title" => "post title",
                    "body" => "post body",
                    "isRead" => false,
                    "comments" => [
                      %{
                        "body" => "comment body",
                        "user" => %{
                          "name" => "username",
                          "age" => 35
                        }
                      }
                    ]
                  }
                ],
                "unreadPosts" => [
                  %{
                    "title" => "post title",
                    "body" => "post body",
                    "isRead" => false,
                    "comments" => [
                      %{
                        "body" => "comment body",
                        "user" => %{
                          "name" => "username",
                          "age" => 35
                        }
                      }
                    ]
                  }
                ]
              }
            ]
          } = data
end

As an exampe, we can have a test that looks like this instead:

test "resolves correctly", %{user: user} do
  query = """
  query {
    users {
      #{document_for(:user, 4)}
    }
  }
  """

  assert_response_matches(query, context: %{current_user: user}) do
    %{
      "users" => [
        %{
          "name" => "username",
          "age" => 35,
          "posts" => [
            %{
              "title" => "post title",
              "body" => "post body",
              "isRead" => false,
              "comments" => [
                %{
                  "body" => "comment body",
                  "user" => %{
                    "name" => "username",
                    "age" => 35
                  }
                }
              ]
            }
          ],
          "unreadPosts" => [
            %{
              "title" => "post title",
              "body" => "post body",
              "isRead" => false,
              "comments" => [
                %{
                  "body" => "comment body"
                  "user" => %{
                    "name" => "username",
                    "age" => 35
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  end
end

This also gives us the added benefit of automatically querying any new fields as they're added to types instead of needing to manually add those new fields in all the tests in which that type is used, further increasing the value of our existing tests!

We can also see in the example above that we're using the assert_response_matches/4 macro which gives us a really nice way to match against the response from our query. This is a pretty small wrapper around the "default" way of testing document resolution shown in the original example, but it gives us a great symmetry between the document and the response and also serves as really great documentation! This way, you see the intended shape of the response clearly in the test, which could make this a valuable test even for non-Elixir developers to use as guidance on how to use this API.

But, in general, by taking this comprehensive approach to testing at this level with the guideline of trying to have at least one of these tests covering every object in your GraphQL schema, you should have some confidence that at the very least, every type in your schema can resolve without error, and it helps us know that if we're using Dataloader, we're able to successfully resolve those associations.

Testing Resolver Functions Using Dataloader

The final part of testing that we'd like to talk about since they are tricky to test is testing resolver functions that use Dataloader's on_load/2 function. They are a bit tricky because these functions return a middleware tuple instead of something a bit easier to test. This means that many people test the behavior in these functions at the document resolution level, but that's not strictly necessary! If you take a look at the tuple that's returned, you'll see the trick to testing those functions.

That function returns a tuple that looks like {:middleware, Absinthe.Middleware.Dataloader, {loader, function}}, and so many folks might expect it to be hard to test, but it's not! If we want to test the actual behavior in that function, which in this case is basically just that Enum.filter/2 call, then we can write our test like this:

test "only returns unread posts" do
  context = MyApp.Schema.context(%{})
  {_, _, {loader, callback}} = resolve_unread_posts(user, nil, context)

  assert {:ok, [%{is_read: false}]} =
    loader
    |> Dataloader.run()
    |> callback.()
end

That's not too bad, right? It just required us to look at the return value from the middleware. All we needed for the test was right there! That callback function that is returned in that middleware tuple is the function that's actually called by Absinthe when resolving the field. Given that the majority of your database access in an Absinthe application should be going through Dataloader, knowing how to use and test functions like this is going to be very helpful as your application develops and more complicated functionality is needed.

Conclusion

We've now seen the three levels of testing that we have at our disposal, how to test at the document resolution level without missing critical pieces of the application, and how to test those tricky functions that use Dataloader. With these three things in mind, testing your Absinthe application should, hopefully, be much easier and more robust.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

The State of Elixir HTTP Clients

Roy Tomeij — 2020-07-28T00:00:00+00:00

In today's post, we'll look at two Elixir HTTP client libraries: Mint and Finch. Finch is built on top of Mint. We'll see the benefits offered by this abstraction layer. We'll also talk about some of the existing HTTP client libraries in the ecosystem and discuss some of the things that make Mint and Finch different. Finally, we'll put together a quick project that makes use of Finch to put all of our learning into action.

Let's jump right in!

What Tools Are Currently Available in the Elixir Ecosystem?

While an HTTP client may not be the most interesting part of your application, more than likely you'll use an HTTP client at some point to interface with 3rd party resources or even internal HTTP microservices. Having an HTTP client with usable ergonomics and a friendly API can help ensure that you deliver application features quickly.

If you are looking for something that comes with the BEAM, you can leverage the :httpc module. While :httpc works great for simple requests, it can be limiting at times, given it doesn't have built-in support for connection pools, SSL verification requires a bit of a ceremony to get set up and the API is not the most intuitive to work with (see this article for more information).

HTTPoison is a 3rd party library that provides some nice abstractions on top of the Erlang library Hackney in order to provide a nice developer experience from within Elixir. It also supports some nice features such as connection pools and the ability to create HTTP client library modules via use HTTPoison.Base. Unfortunately, in the past, I have experienced issues with Hackney in high traffic applications (so have other users). If I suspect that I will be making a large number of concurrent requests to an HTTP API, I try to keep that in mind and plan for possible failure scenarios.

Tesla provides additional abstraction layers in the form of an HTTP client that can be configured using middlewares (similar to Plug), mock responses for testing, and even different adapters to perform the HTTP requests. In fact, you can even use Mint as an adapter in Tesla. If I'm making a very involved HTTP client with many different interactions and behaviors, I'll usually reach for Tesla as it makes it easy to package these pieces of functionality together, and the testing utilities make your ExUnit tests very clean.

How Are Mint and Finch Different?

The previously mentioned tools all rely on a process to keep track of the ongoing HTTP connection. Mint on the other hand provides a low-level, process-less API for interacting with TCP/SSL sockets. Every time you interact with Mint, for example, you'll be given back a new Mint.HTTP1 or Mint.HTTP2 struct handler. In addition, any data coming into the socket will be sent as a message to the process that initiated the connection. This data can then be captured via a simple receive block and handled accordingly. While this may seem limiting, it is by design. The architecture of Mint lends itself to being extensible and enables other library authors to wrap Mint however they see fit.

This is where Finch comes in. Finch is a library that wraps Mint and provides many of the HTTP client features that you would expect from a fully-fledged HTTP client. For example, with Finch, you get connection pooling and request Telemetry out-of-the-box. While that feature set may not be as thorough as HTTPoison or Tesla, Finch is very much focused on being lightweight and performant.

Hands-On Project

Now that we've discussed some of the design decisions that went into Mint and Finch, it's time to dive into a sample project. Our sample application will be a functional programming language Hacker News counter. It will work by taking a Hacker News article ID and then fetching all the child posts of that parent post. It will then leverage a connection pool to the Hacker News Firebase API to perform a number of concurrent calls to the API to fetch all the child posts. Once all the child posts have been fetched, all the text bodies will be extracted and analyzed for the names of certain functional programming languages. Finally, we'll print out our results, along with some metrics that were collected via Telemetry events. With all that being said, let's jump right into it (for reference, all the code can be found at https://github.com/akoutmos/functional_langs)!

Let's start off by creating a new Elixir project with a supervision tree:

$ mix new functional_langs --sup

With that in place, let's open the mix.exs file and add the required dependencies:

defp deps do
  [
    {:finch, "~> 0.3.0"},
    {:jason, "~> 1.2"}
  ]
end

Once your mix.exs file has been updated, switch over to the terminal and run mix deps.get to fetch your dependencies. Next, we'll want to create the file lib/functional_langs/hacker_news_client.ex that will encompass all of our Finch related code. This file will include all the calls necessary to interact with the Hacker News Firebase API and will also include some utility functions to extract the desired data from the JSON payloads. Let's start off by adding some of the foundation of our lib/functional_langs/hacker_news_client.ex file:

defmodule FunctionalLangs.HackerNewsClient do
  alias Finch.Response

  def child_spec do
    {Finch,
     name: __MODULE__,
     pools: %{
       "https://hacker-news.firebaseio.com" => [size: pool_size()]
     }}
  end

  def pool_size, do: 25

  def get_item(item_id) do
    :get
    |> Finch.build("https://hacker-news.firebaseio.com/v0/item/#{item_id}.json")
    |> Finch.request(__MODULE__)
  end
end

Our child_spec/0 function defines the child spec for the Finch connection pool. We will be leveraging this function in our lib/functional_langs/application.ex file so that we can start up our Finch connection pool within our application supervision tree. Our pool_size/0 function defines how big our connection pool to the https://hacker-news.firebaseio.com address will be. For testing purposes, 25 concurrent connections should be more than enough to traverse even the largest Hacker News posts. Lastly, the get_item/1 function is what makes the actual GET call to the Hacker News Firebase API. We first define the HTTP verb as an atom, we then build the request, and finally, we make the request while providing the name of the module (notice that this lines up with the name of the connection pool in child_spec/0). With that in place, we are able to make calls to the Hacker News Firebase API and leverage our connection pool using Finch.

With that in place, let's wrap up our FunctionalLangs.HackerNewsClient module:

defmodule FunctionalLangs.HackerNewsClient do
  ...

  def get_child_ids(parent_item_id) do
    parent_item_id
    |> get_item()
    |> handle_parent_response()
  end

  defp handle_parent_response({:ok, %Response{body: body}}) do
    child_ids =
      body
      |> Jason.decode!()
      |> Map.get("kids")

    {:ok, child_ids}
  end

  def get_child_item(child_id) do
    child_id
    |> get_item()
    |> get_child_item_text()
  end

  defp get_child_item_text({:ok, %Response{body: body}}) do
    body
    |> Jason.decode!()
    |> case do
      %{"text" => text} -> String.downcase(text)
      _ -> ""
    end
  end
end

While the majority of this code snippet is related to unpacking and massaging the data, one thing I'll touch on though is the pattern match on the Response struct. If you recall from the previous code snippet, we alias Finch.Response and then use that alias here. By pattern matching on the struct, we are able to easily extract the response body and work with the returned data. For more details on Finch.Response struct, feel free to check out the documentation.

With our client module wrapped up, let's quickly open up lib/functional_langs/application.ex and add the following line to our supervision tree so that our connection pool can be started with the application:

def start(_type, _args) do
  children = [
    FunctionalLangs.HackerNewsClient.child_spec()
  ]

  ...
end

Our application will use the Hacker News client that we just wrote to fetch all the child posts of a Hacker News item and then look for occurrences of functional programming language names in each of those child posts. Once we have everything tallied up, we'll present a nice ASCII chart and some overarching metrics. With that said, let's open up lib/functional_langs.ex and start by adding the following:

defmodule FunctionalLangs do
  require Logger

  alias FunctionalLangs.HackerNewsClient

  @langs_of_interest ~w(elixir erlang haskell clojure scala f# idris ocaml)
  @label_padding 7
  @telemetry_event_id "finch-timings"

  def generate_report(parent_item_id) do
    # Setup Telemetry events and Agent to store timings
    {:ok, http_timings_agent} = Agent.start_link(fn -> [] end)
    start_time = System.monotonic_time()
    attach_telemetry_event(http_timings_agent)

    # Get all of the child IDs associated with a parent item
    {:ok, child_ids} = HackerNewsClient.get_child_ids(parent_item_id)

    # Concurrently process all of the child IDs and aggregate the results into a graph
    child_ids
    |> Task.async_stream(&HackerNewsClient.get_child_item/1, max_concurrency: HackerNewsClient.pool_size())
    |> Enum.reduce([], fn {:ok, text}, acc ->
      [text | acc]
    end)
    |> Enum.map(&count_lang_occurences/1)
    |> Enum.reduce(%{}, &sum_lang_occurences/2)
    |> print_table_results()

    # Calculate average API request time and total time
    average_time = calc_average_req_time(http_timings_agent)
    total_time = System.convert_time_unit(System.monotonic_time() - start_time, :native, :millisecond)

    # Clean up side-effecty resources
    :ok = Agent.stop(http_timings_agent)
    :ok = :telemetry.detach(@telemetry_event_id)

    IO.puts("Average request time to Hacker News Firebase API: #{average_time}ms")
    IO.puts("Total time to fetch all #{length(child_ids)} child posts: #{total_time}ms")
  end
end

The first few lines of generate_report/1 are responsible for starting an Agent that will be used to collect Telemetry metrics from Finch. The Agent PID is sent over to the attach_telemetry_event/1 function so that the handler that is created knows the PID of the Agent. The implementation of the attach_telemetry_event/1 function is as follows:

defmodule FunctionalLangs do
  ...

  defp attach_telemetry_event(http_timings_agent) do
    :telemetry.attach(
      @telemetry_event_id,
      [:finch, :response, :stop],
      fn _event, %{duration: duration}, _metadata, _config ->
        Agent.update(http_timings_agent, fn timings -> [duration | timings] end)
      end,
      nil
    )
  end
end

This function simply attaches to the Finch [:finch, :response, :stop] event and adds the duration metric to the Agent's list of metrics. Back in our generate_report/1 function, our next step is to get all the child item IDs from the provided parent_item_id and put that into our processing pipeline. Our processing pipeline leverages Task.async_stream/3 in order to concurrently process all the child item IDs. One important thing to note is that our options to Task.async_stream/3 include max_concurrency: HackerNewsClient.pool_size(). The reason for this is that we only want to run the same number of concurrent tasks as there are connections in the Finch connection pool.

Our next piece of the pipeline is to reduce on the results from Task.async_stream/3 and to extract all the text blocks that were fetched. Afterward, we perform an Enum.map/2 and an Enum.reduce/3 on those results in order to tally up our results. The implementations of the functions used in Enum.map/2 and Enum.reduce/3 are:

defmodule FunctionalLangs do
  ...

  defp count_lang_occurences(text) do
    Map.new(@langs_of_interest, fn string_of_interest ->
      count = if String.contains?(text, string_of_interest), do: 1, else: 0

      {string_of_interest, count}
    end)
  end

  defp sum_lang_occurences(counts, acc) do
    Map.merge(acc, counts, fn _lang, count_1, count_2 ->
      count_1 + count_2
    end)
  end
end

The last step in the pipeline is to take the results and pretty print a sorted ASCII chart so that we can see the results from greatest to least. Below is the implementation of print_table_results/1:

defmodule FunctionalLangs do
  ...

  defp print_table_results(results) do
    results
    |> Enum.sort(fn {_lang_1, count_1}, {_lang_2, count_2} ->
      count_1 > count_2
    end)
    |> Enum.each(fn {language, count} ->
      label = String.pad_trailing(language, @label_padding)
      bars = String.duplicate("█", count)

      IO.puts("#{label} |#{bars}")
    end)
  end
end

With that completed, all that's left is to aggregate all the captured Telemetry metrics and compute the average request time, clean up our side-effecty resources and print the results. The implementation of calc_average_req_time/1 looks like this:

defmodule FunctionalLangs do
  ...

  defp calc_average_req_time(http_timings_agent) do
    http_timings_agent
    |> Agent.get(fn timings -> timings end)
    |> Enum.reduce({0, 0}, fn timing, {sum, count} ->
      {sum + timing, count + 1}
    end)
    |> case do
      {_, 0} ->
        "0"

      {sum, count} ->
        sum
        |> System.convert_time_unit(:native, :millisecond)
        |> Kernel./(count)
        |> :erlang.float_to_binary(decimals: 2)
    end
  end
end

With all that in place, we are ready to use our application! In your terminal, run iex -S mix to launch an IEx session with all of our modules loaded. Once your IEx session is up and running, you can call your FunctionalLangs.generate_report/1 function to generate the language occurrence report (the "23702122" is from the "July 2020 Who is hiring" post):

iex(1) ▶ FunctionalLangs.generate_report("23702122")
scala   |█████████████████████████████████████████████
elixir  |██████████████
clojure |██████████
haskell |█████
f#      |███
erlang  |██
ocaml   |█
idris   |█
Average request time to Hacker News Firebase API: 58.17ms
Total time to fetch all 564 child posts: 2185ms

Conclusion

Thanks for sticking with me to the end and hopefully you learned a thing or two about Mint and Finch! From this tutorial, we learned how to create connection pools with Finch, attach Telemetry handlers to Finch events, and how to make large amounts of concurrent requests to an API. While Finch is relatively new compared to other HTTP clients, it is built upon tried and tested libraries like Mint and NimblePool.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Building State Machines in Elixir with Ecto

Roy Tomeij — 2020-07-14T00:00:00+00:00

Among the many useful patterns in computer science, there is the concept of a Finite-state machine (FSM).

It's a great abstraction in many different scenarios, where you want to model a certain process that goes through a predefined set of states, with different behaviors, depending on what state it is in.

In this post, you'll learn how to implement this pattern with Elixir's Ecto and when to use it.

Use Cases

When you model a long-running flow that requires multiple steps, and where each step has different requirements, a state machine can be a good choice as an abstraction. A few examples:

A user onboarding flow where the user first signs up, then adds some extra required info, confirms his email, then enables 2FA, and only then is allowed into the system
A shopping cart which starts out empty, allows products to be added indefinitely and can proceed to payment/shipping if enough products are added
A task in a project management pipeline. e.g: Tasks start out in the backlog, can be assigned to people, moved to "in progress", and later to "done"

An Example of a Finite-State Machine

For this post, we'll stick with a small example that illustrates the flow of a state machine: A door.

A door can be locked or unlocked. It can also be opened or closed. While unlocked, it can also be opened.

We could model this as a finite-state machine, such as the following:

This FSM has:

3 possible states: Locked, Unlocked, Opened
4 possible transitions or events: Unlock, Open, Close, Lock

It can be inferred from the diagram that it's impossible to transition from Locked to Opened. Or in plain words: you need to unlock the door first. The state machine diagram describes the behavior. But how can we go about implementing it?

State Machines as Elixir Processes

Since OTP 19, Erlang provides a :gen_statem module that allows implementing gen_server-like processes that behave as state machines (where the current state influences their behavior). Let's see what that would look like for our door example:

defmodule Door do
  @behaviour :gen_statem

  def start_link do
    :gen_statem.start_link( __MODULE__,:ok,[] )
  end

  @impl :gen_statem
  def init(_), do: {:ok, :locked, nil}

  @impl :gen_statem
  def callback_mode, do: :handle_event_function

  @impl :gen_statem
  def handle_event({:call, from}, :unlock, :locked, data) do
    {:next_state, :unlocked, data, [{:reply, from, {:ok, :unlocked}}]}
  end

  def handle_event({:call, from}, :lock, :unlocked, data) do
    {:next_state, :locked, data, [{:reply, from, {:ok, :locked}}]}
  end

  def handle_event({:call, from}, :open, :unlocked, data) do
    {:next_state, :opened, data, [{:reply, from, {:ok, :opened}}]}
  end

  def handle_event({:call, from}, :close, :opened, data) do
    {:next_state, :unlocked, data, [{:reply, from, {:ok, :unlocked}}]}
  end

  def handle_event({:call, from}, _event, _content, data) do
    {:keep_state, data, [{:reply, from, {:error, "invalid transition"}}]}
  end
end

This implements a process that starts out in the :locked state. By sending appropriate events, we are able to match the current state with the transition requested and perform the required transformations. An additional data argument is kept for any additional state that is needed, but we're not using that in this case.

To use this process, you can call it with the desired transition that you want to execute. If the current state allows that transition, it will work. Otherwise, an error is returned (due to the last, catch-all match of the code snippet).

{:ok, pid} = Door.start_link()

:gen_statem.call(pid, :unlock)
# {:ok, :unlocked}

:gen_statem.call(pid, :open)
# {:ok, :opened}

:gen_statem.call(pid, :close)
# {:ok, :closed}

:gen_statem.call(pid, :lock)
# {:ok, :locked}

:gen_statem.call(pid, :open)
# {:error, "invalid transition"}

If our state machine is more data-oriented than process-oriented, we may want to go with a different approach...

State Machines as Ecto Models

There are a couple of Elixir packages that deal with this problem. For this post, I'll be using fsmx, but other packages such as machinery also provide similar features.

This package allows us to model the same kind of states and transitions within an existing Ecto Model:

defmodule PersistedDoor do
  use Ecto.Schema

  schema "doors" do
    field :state, :string, default: "locked"
    field :terms_and_conditions, :boolean
  end

  use Fsmx.Struct, transitions: %{
    "locked" => "unlocked",
    "unlocked" => ["locked", "opened"],
    "opened" => "unlocked"
  }
end

We can see that Fsmx.Struct receives all possible transitions as an argument. This allows it to check for unwanted transitions and prevent them from happening. Now, we can transition using a traditional, non-Ecto approach:

door = %PersistedDoor{state: "locked"}

Fsmx.transition(door, "unlocked")
# {:ok, %PersistedDoor{state: "unlocked", color: nil}}

But we can also ask for the same in the form of an Ecto changeset:

# get an existing door from the Database
door = PersistedDoor |> Repo.one()

Fsmx.transition_changeset(door, "unlocked")
|> Repo.update()

This changeset only updates the :state field. But we can extend it to include additional fields, as well as validations. Let's say that, in order to open a door, we need to accept its terms & conditions:

defmodule PersistedDoor do
  # ...

  def transition(changeset, _from, "opened", params) do
    changeset
    |> cast(params, [:terms_and_conditions])
    |> validate_acceptance(:terms_and_conditions)
  end
end

Fsmx looks for an optional transition_changeset/4 function in your schema and calls it with the previous state as well as the next one. You can pattern match on those to add specific conditions for each transition.

Dealing With Side Effects

It's one thing to transition the state machine itself and move forward with the state. But another big benefit of state machines is the ability to deal with particular side effects that come out of each state.

Let's say, for example, you want to get notified every time someone unlocks your door. You might want to trigger an email when it happens. But you want these two operations to be a single, atomic operation.

Ecto deals with atomicity via Ecto.Multi which groups multiple operations inside a database transaction. It also has an Ecto.Multi.run/3 function that allows you to run arbitrary code within that same transaction.

Fsmx in turn, integrates with Ecto.Multi by providing you with a way to run state transitions as part of an Ecto.Multi, while also providing you an additional callback that is executed in that case:

defmodule PersistedDoor do
  # ...

  def after_transaction_multi(changeset, _from, "unlocked", params) do
    Emails.door_unlocked()
    |> Mailer.deliver_later()
  end
end

Now, you can execute a transition as shown:

door = PersistedDoor |> Repo.one()

Ecto.Multi.new()
|> Fsmx.transition_multi(schema, "transition-id", "unlocked")
|> Repo.transaction()

This transaction will use the same transition_changeset/4 from above to compute the necessary changes to the model and will include the new callback as an Ecto.Multi.run call. The result is that an email is sent (asynchronously, using Bamboo, so as not to run within the transaction itself). If the changeset is invalidated for some reason, the email ends up never being sent, resulting in an atomic execution of both operations.

Conclusion

Next time you're modeling some kind of stateful behavior, consider looking into a state-machine approach. Regardless of which flavor you use, the ability to translate a concrete diagram to actual code and the ability to test each state, transition, and side effect as a separate piece is a huge advantage.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Best Practices for Background Jobs in Elixir

Roy Tomeij — 2020-06-24T00:00:00+00:00

Erlang & Elixir are ready for asynchronous work right off the bat. Generally speaking, background job systems aren't needed as much as in other ecosystems but they still have their place for particular use cases.

This post goes through a few best practices I often try to think of in advance when writing background jobs, so that I don't hit some of the pain points that have hurt me multiple times in the past.

If you've ever deployed a new task, only to find out that it has gone rogue with a bug that caused it to misbehave (e.g.: sending way too many emails, way too quickly), you may have gone through similar bugs as well.

Flavours

Elixir already gives you the ability to schedule asynchronous work pretty easily. Something as simple as this already covers a lot:

Task.async(fn ->
  # some heavy lifting
end)

You might need something a bit more powerful, either just for convenience (having some tooling & monitoring around that task), or because you need something like periodic jobs. Again, all this can be achieved with something like a GenServer:

defmodule PeriodicJob do
  use GenServer

  @period 60_000

  def init do
    Process.send_after(self(), :poll, @period)

    {:ok, :state}
  end

  def handle_info(:poll, state) do
    # some heavy lifting

    Process.send_after(self(), :poll, @period)
  end
end

You can also use a job queuing library such as Quantum. If you come from Ruby land and are used to libraries such as Sidekiq, you might be more familiar with something like this:

#
# lib/my_app/scheduler.ex
#
defmodule MyApp.Scheduler do
  use Quantum.Scheduler, opt_app: :my_app
end

#
# config/config.exs
#
config :my_app, MyApp.Scheduler,
  jobs: [
    first: [
      # every hour
      schedule: "0 * * * *",
      task: {MyApp.ExampleJob, :run, []}
    ],
    second: [
      # every minute
      schedule: "* * * * *",
      task: {MyApp.AnotherExampleJob, :run, []}
    ]
  ]

Some may argue that since Erlang/OTP already provides the infrastructure for creating these processes, packages such as Quantum are not necessary. However, the structure created by them can end up being more intuitive, especially if you're not that familiar with OTP. This might be the case with someone coming from Ruby or other such communities.

How to Structure Background Jobs

Let's now get into a few tips that will help you keep your jobs ready to deal with potential future problems!

Most of them are preventive measures due to the fact that all of these are background processes. They're not responding to an HTTP request and they happen without any intervention, thus sometimes, debugging can be hard if you don't take some precautions.

Let's consider a small example that sends confirmation emails to users that haven't received it yet:

defmodule MyApp.ExampleJob do
  def run do
    get_users()
    |> Enum.each(fn user ->
      # send single email to user
    end)
  end

  defp get_users do
    MyApp.User
    |> where(confirmation_email_sent: false)
    |> MyApp.Repo.all()
  end
end

1. Put in a Kill Switch

This is one of those mistakes I'll never make again since it has hurt me so many times.

Let's say you've created a background job, tested, deployed, and configured it to run periodically and send some emails.

It hits production, and you soon notice that something's wrong. The same 100 people are being spammed with emails every minute. You messed up the geth_next_batch/1 function, and it always goes over the same batch of users. It's a developer's horror story. You need to fix it (or kill it) quickly, but all that time waiting for a new release to get online is physically painful.

So, avoid that:

defmodule MyApp.ExampleJob do
  def run do
    return if !enabled?

    # ...
  end

  defp enabled? do
    # check a Redis flag, or a database record, or anything really
  end
end

You can plug in some persistent system that allows you to quickly toggle the job on/off. A good suggestion would be to use a feature flag package, such as FunWithFlags.

2. Always Batch Your Jobs by Default

It's easy to miss this one on a first draft. You're just trying to quickly get something online. But in some cases, it may be important to not hurt your performance if you're working on a very resource-intensive job, or simply, if your list of records to process grows too quickly.

Doing User |> where(confirmation_email_sent: false) |> Repo.all() can be dangerous if there's potential for that to yield too many results. You may end up consuming too many resources for something that could be done in smaller batches, keeping your system a lot more stable:

defmodule MyApp.ExampleJob do
  @batch_size 100

  defp get_users do
    MyApp.User
    |> where(confirmation_email_sent: false)
    |> limit(^@batch_size)
    |> MyApp.Repo.all()
  end
end

Whatever job queue mechanism you plug this worker into, it will end up being called frequently. So you shouldn't hurry in processing smaller batches one at a time.

3. Avoid Overlaps

This is kind of related to the previous point, but it's a concern that goes beyond performance.

If you program a job to run every minute, and a single execution has the potential to last longer than that, you end up risking cascading performance problems, or even worse, race conditions, where the first and second executions are both trying to process the same set of data, and conflict with each other in the process.

This is obviously dependent on what your exact business logic is, but as a general rule, it's best to be defensive here.

If you use a GenServer approach like the one showcased above, this is solved automatically, as instead of scheduling jobs every minute, you can instead use Process.send_after(self(), :poll, delay) to only schedule the next run after the current one has finished, avoiding overlap.

When using Quantum, you also have an overlap: true option that you can add to automatically prevent this:

config :my_app, MyApp.Scheduler,
  jobs: [
    first: [
      # every hour
      schedule: "0 * * * *",
      task: {MyApp.ExampleJob, :run, []}
      overlap: false
    ]

This, by the way, might already be reason enough to consider using a package rather than just plain Elixir.

4. Plug in a Manual Mode

If your job is processing a batch of records, it's useful to plug in some public functions that allow you to manually process specific records. This can serve two purposes:

Better ability to debug the job
Ability to do a few manual runs before enabling the global job (by toggling the feature flag discussed above)

A sample structure could look like this:

defmodule MyApp.ExampleJob do
  import MyApp.Lock

  def run do
    lock("example_job", fn ->
      get_users()
      |> Enum.each(&process_user/1)
    end)
  end

  def run_manually(users) when is_list(users) do
    lock("example_job", fn ->
      users
      |> Enum.each(&process_user/1)
    end)
  end

  def run_manually(user), do: run_manually([user])

  def process_user(%User{} = user) do
    # process a single user
  end
end

In this case, we're creating a run_manually/1 public function that can receive either a single user or a batch of them and performs the same logic as the automatic job would.

One important detail here is to again avoid a race condition, which in this case, is being done with a custom Lock module that uses the redis_mutex package to prevent potential issues:

defmodule MyApp.Lock do
  use RedisMutex
  require Logger

  def lock(lock_name, fun) do
    with_lock(lock_name, 60_000) do
      fun.()
    end
  rescue
    _e in RedisMutex.Error ->
      Logger.debug("#{locker_name} another process already running")
  end
end

The lock, which is invoked both on manual runs as well as the regular background job, ensures that you won't cause any unintentional conflicts if you try to do a manual run at the same time the job is doing the same processing. It also happens to solve the overlap problem discussed previously in this post.

Conclusion

All these tips come from something that I bumped into in the past, usually related to production bugs or user complaints. So I hope some of them help you avoid the same mistakes. Let me know if you have any further thoughts! 👋

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Using Mnesia in an Elixir Application

Roy Tomeij — 2020-05-19T00:00:00+00:00

This post was updated on 9 August 2023 with some minor code changes.

In today's post, we'll learn about Mnesia, see when you would use such a tool, and take a look at some of the pros and cons of using it. After covering the fundamentals of Mnesia, we'll dive right into a sample application where we'll build an Elixir application that uses Mnesia as its database. Let's jump right in!

Introduction to Mnesia

At a high level, Mnesia is a Database Management System (DBMS) that is baked into OTP. Thus, if you are using Elixir or Erlang, you have the ability to leverage Mnesia out-of-the-box. No additional dependencies need to be installed and no separate systems need to be running. Before considering migrating everything from your existing database to Mnesia, let's discuss what Mnesia was designed for and what problems it aims to solve.

Mnesia was largely designed to solve the problems that existed in the telecommunications problem space. Specifically, some of the following requirements needed to be fulfilled:

Fast key/value lookup times where you need soft real-time latency guarantees. A soft real-time system is one where the system should be able to service the majority of its requests within a given time frame and a failure to do so generally means degradation of service (i.e the data is no longer useful after the time frame has passed). A hard real-time system, on the other hand, is a system that must respond within a given time frame or else it is considered a system failure.
The ability to perform complex queries (like you would in SQL for example), but without soft real-time latency guarantees
A high level of fault tolerance

In a typical DBMS, your application would need to either make a network call to a separate machine where the database is running, or it would have to connect to the database process that is running on the same machine. Either way, the data that is contained within that database resides in an entirely separate memory space than the application, and therefore, there is an inescapable amount of latency overhead.

On the other hand, Mnesia runs within the same memory space as the application. As a result of being baked into the language and runtime, you are able to fetch data out of Mnesia at soft real-time speeds. In other words, your application and database are running side-by-side and there is little to no communication overhead between the two.

Another important thing to note is that Mnesia stores all of the Erlang data types natively, and so there is no need to marshall/unmarshall data when you read/write to Mnesia (marshalling is the process of converting data from one format to another for the purposes of storing it or transmitting it).

When performing complex queries against your Mnesia database, you can either leverage Query List Comprehensions (QLC) or you can write Match Specifications. In addition, you can also add indexes to your Mnesia tables for fields that you know you'll be querying often. Using these tools, you can perform arbitrary queries against your tables and extract the relevant data.

A primary requirement of telecommunications systems is that they must be running nonstop. Downtime means missed or dropped calls. Mnesia addresses this by allowing tables to be replicated across the various nodes in the cluster.

When running within a transaction, the data that needs to be committed must be written to all the configured table replicas. If nodes are unavailable during a write, the transaction will update the available node replicas and will update the unavailable node replicas when they come back online. Through this replication mechanism, Mnesia is able to provide a high level of fault tolerance.

Mnesia and the CAP Theorem

You may be wondering exactly where it falls in regards to the CAP theorem. For those unfamiliar with the CAP theorem, it basically states that, when dealing with distributed systems, you have three characteristics at play but can only guarantee two at any given time. Those three characteristics are:

Consistency: Whenever a read is made against your database, the database will respond with the most recently updated data.
Availability: Whenever a request is made against your database, the database will respond with some data even if it's out of date (i.e. newer data has been committed but has not propagated to all nodes).
Partition tolerance: Whenever a request is made against your database, it will be able to respond regardless of some nodes being unavailable.

When a network partition does occur (i.e. some database nodes are unavailable), your system must make a trade-off. Does it favor consistency and error out on any requests while some nodes are unavailable, or does it favor availability by servicing the request with the understanding that there may be some data inconsistency when the missing nodes come back online?

Given that Mnesia will propagate transaction commits across all table replicas and does not support any kind of eventual consistency, it is more of a CP style database. In the case of a network partition where the separate partitions are both handling requests, the application will need to deal with reconciliation of the data.

When To Use Mnesia Over PostgreSQL or Other Database

Like many things in Software Engineering and Systems Design, it's all about making the correct trade-offs. Whether Mnesia is right or not for your application largely depends on its requirements. Personally, I have used Mnesia in production primarily to support some soft real-time use cases with very good results.

The data that was stored in Mnesia was needed only for the duration of the user's session and would then get cleared after the user's interaction with the system ceased. Thus, there wasn't a lot of pressure on system resources (RAM specifically, as the tables need to fit into RAM), as the size of the tables would reflect the number of users actively using the system. For situations where you need to store a large amount of data and you do not require soft real-time response times, a traditional DBMS such as MySQL or Postgres may be a better choice.

For situations where you see yourself reaching for Redis or Memcached, you may want to consider looking into Mnesia, given that it fills a similar need and is built into OTP. For more information regarding this topic, I would suggest looking at Mnesia docs.

Hands-on Project with Mnesia

In order to get familiar with Mnesia, we'll be creating a very simple banking application that leverages Mnesia as its database. While we could leverage the Mnesia API directly via :mnesia, we will instead opt to use the Amnesia library as it provides a nice Elixir wrapper around the Mnesia API. Our banking application will support the following operations:

Create new accounts
Transfer money between accounts
Fetch account details
Deposit funds into an account
Withdraw funds from an account
Search for accounts with a low balance

Let's create a new Elixir project using the following terminal command:

$ mix new fort_knox --sup

Quick tip: The --sup flag adds a supervision tree to the app.

After creating the Elixir project, open up the mix.exs file and add amnesia to it as follows:

defp deps do
  [
    {:amnesia, "~> 0.2.8"}
  ]
end

After that has been done, you can run mix deps.get to fetch the amnesia dependency. Next, we'll want to create a module that defines all the table schemas in our Mnesia database. For our sample application, we will only have one table defined for bank accounts. To do this, add the following content to lib/database.ex

use Amnesia

defdatabase Database do
  deftable(
    Account,
    [{:id, autoincrement}, :first_name, :last_name, :balance],
    type: :ordered_set,
    index: [:balance]
  )
end

Our database contains only the Account table and specifies that it has 3 fields along with an auto-incrementing id field. With the database definition in place, let's go back to our terminal and run the following command:

$ mix amnesia.create -d Database --disk

After executing that command, you will notice that a new directory (Mnesia.nonode@nohost) has been created for us at the root of our project. This directory contains all the disk persisted data so that our data can be maintained across application restarts. To delete all of the persisted database data, you can either rm -rf Mnesia.nonode@nohost or run mix amnesia.drop -d Database --schema.

With that in place, it's time to work on some of our business logic. Let's create a file at lib/fort_knox/accounts.ex and start off by creating functions that will create a new account and fetch existing accounts:

# lib/fort_knox/accounts.ex

defmodule FortKnox.Accounts do
  require Amnesia
  require Amnesia.Helper
  require Exquisite
  require Database.Account

  alias Database.Account

  def create_account(first_name, last_name, starting_balance) do
    Amnesia.transaction do
      %Account{first_name: first_name, last_name: last_name, balance: starting_balance}
      |> Account.write()
    end
  end

  def get_account(account_id) do
    Amnesia.transaction do
      Account.read(account_id)
    end
    |> case do
      %Account{} = account -> account
      _ -> {:error, :not_found}
    end
  end
end

Our module begins with a few require statements to pull in Amnesia functionality. We can then leverage Account as a struct to conveniently interact with the Account table in Mnesia. To create a new Account entry in the table, we create the struct and call Account.write() within a transaction. If you do not want to perform your database actions within a transaction, you can also leverage the dirty read/write API calls, but that is not recommended. When looking up existing accounts by their id, we once again leverage a transaction and match on an Account struct if an account was found. Let's go ahead and add the remainder of our functionality in lib/fort_knox/accounts.ex:

# lib/fort_knox/accounts.ex

defmodule FortKnox.Accounts do
  ...

  def transfer_funds(source_account_id, destination_account_id, amount) do
    Amnesia.transaction do
      accounts = {Account.read(source_account_id), Account.read(destination_account_id)}

      case accounts do
        {%Account{} = source_account, %Account{} = destination_account} ->
          if amount <= source_account.balance do
            adjust_account_balance(destination_account, amount)
            adjust_account_balance(source_account, -amount)
            :ok
          else
            {:error, :insufficient_funds}
          end

        {%Account{}, _} ->
          {:error, :invalid_destination}

        {_, _} ->
          {:error, :invalid_source}
      end
    end
  end

  def get_low_balance_accounts(min_balance) do
    Amnesia.transaction do
      Account.where(balance < min_balance)
      |> Amnesia.Selection.values()
    end
  end

  def deposit_funds(account_id, amount) do
    Amnesia.transaction do
      case Account.read(account_id) do
        %Account{} = account ->
          adjust_account_balance(account, amount)

        _ ->
          {:error, :not_found}
      end
    end
  end

  def withdraw_funds(account_id, amount) do
    Amnesia.transaction do
      case Account.read(account_id) do
        %Account{} = account ->
          if amount <= account.balance do
            adjust_account_balance(account, -amount)
          else
            {:error, :insufficient_funds}
          end

        _ ->
          {:error, :not_found}
      end
    end
  end

  defp adjust_account_balance(%Account{} = account, amount) do
    account
    |> Map.update!(:balance, &(&1 + amount))
    |> Account.write()
  end
end

The withdraw_funds/2, deposit_funds/2 and transfer_funds/3 functions should be relatively straight forward as they are a mixture of reads and writes to update accounts within a transaction. The get_low_balance_accounts/1 will probably seem new as we have a where clause to query our database records. The Exquisite library (which Amnesia depends on) provides the ability to generate Mnesia Match Specifications which are used to perform custom queries [5].

With all that in place, let's take this all for a test drive. We'll first seed our database with some initial accounts and then transfer some funds between the accounts. Open up an IEx session via iex -S mix and type the following:

iex(1) ▶ [
...(1) ▶ {"Josh", "Smith", 1_000},
...(1) ▶ {"Tom", "Lee", 500},
...(1) ▶ {"Joe", "Diaz", 1_500}
...(1) ▶ ] |>
...(1) ▶ Enum.each(fn {first_name, last_name, amount} ->
...(1) ▶ FortKnox.Accounts.create_account(first_name, last_name, amount)
...(1) ▶ end)
:ok

iex(2) ▶ FortKnox.Accounts.get_account(1)
%Database.Account{balance: 1000, first_name: "Josh", id: 1, last_name: "Smith"}

iex(3) ▶ FortKnox.Accounts.get_account(2)
%Database.Account{balance: 500, first_name: "Tom", id: 2, last_name: "Lee"}

iex(4) ▶ FortKnox.Accounts.transfer_funds(2, 1, 400)
:ok

iex(5) ▶ FortKnox.Accounts.get_account(1)
%Database.Account{balance: 1400, first_name: "Josh", id: 1, last_name: "Smith"}

iex(6) ▶ FortKnox.Accounts.get_account(2)
%Database.Account{balance: 100, first_name: "Tom", id: 2, last_name: "Lee"}

iex(7) ▶ FortKnox.Accounts.get_low_balance_accounts(250)
[%Database.Account{balance: 100, first_name: "Tom", id: 2, last_name: "Lee"}]

After running all of these commands, feel free to quit from IEx via Ctrl+C and go back to using iex -S mix. If you run Database.Account.count(), you'll see that we get a value of 3 since our data persisted across IEx sessions and was not destroyed.

Conclusion

Thanks for sticking with me to the end and hopefully you learned a thing or two about Mnesia and how to go about using it within an Elixir application. Regardless of whether you decide to use Mnesia in a production context or not, I would highly suggest at least experimenting with it so as to better appreciate the amazing things that you get out-of-the-box with OTP.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Elixir Alchemy by AppSignal

Debugging Slow Ecto Queries with AppSignal

Finding the Slow Query in AppSignal

Understanding Why It's Slow

Common Ecto Fixes

Inefficient Queries in Ecto

The N+1 Query Problem

Verifying Our Changes Worked

Wrap-Up

Domains and Resources in Ash for Elixir

What Is Ash for Elixir?

The AshTherapy Use Case

Check Out the Code

Ash Domains

Understanding Ash Resources

Resource Structure

Attributes

Declaring Relationships

Declaring Business Actions

Reading Data with Built-in Preloading

Declare Once, Derive the Rest

Quick Comparison: Ecto vs Ash

Playing with Ash in IEx

Understanding Ash.create and Ash.read

Create a User

Start a Conversation

Send a Message

Fetch the Conversation Messages

Wrapping Up

AppSignal’s Top 5 Elixir Posts in 2025

Top 5 Elixir Blog Posts in 2025 ⚗️

Building a Distributed Rate Limiter in Elixir with HashRing

Getting Started with Dialyzer in Elixir

Structs and Embedded Schemas in Elixir: Beyond Maps

Advanced Debugging in Elixir with IO.inspect

Batch Updates and Advanced Inserts in Ecto for Elixir

Season's Greetings ❆ ⛄

Build an Elixir App with Cowboy

Why Cowboy for Elixir?

Project Setup and Bootstrapping Cowboy

Topic Model, Publish Flow, and Subscriptions

Replay, Stats, and Testing

When to Use Cowboy vs. Phoenix for Elixir

Cowboy Shines When…

Phoenix Makes More Sense When…

Wrapping Up

Debugging in Elixir with Observer

Set Up: Verifying Observer in Your Elixir Installation

The Observer GUI in Erlang for Elixir

Project Setup: Elixir App with Leaking Memory

Creating the Project

Add the Offending Process Module

Update the Application Supervisor

Add the Observer Backend

Connecting Observer from a Separate Node

Debugging a Remote System with Observer

Going Further

Wrapping Up

Monitor the Performance of Your Ecto for Elixir App with AppSignal

Prerequisites

Setting Up AppSignal for Elixir

Understanding Ecto Performance Metrics

Query Execution Time and Throughput

N+1 Queries

Beyond Basics — Monitoring Ecto for Elixir with AppSignal Custom Instrumentation

Monitoring Ecto Batch Inserts

Why Use Custom Instrumentation for Your Elixir App?

Custom Instrumenting Ecto Batch Inserts

Wrapping Up

Batch Updates and Advanced Inserts in Ecto for Elixir

Prerequisites

Introducing Ecto for Elixir

Ecto's Architecture

How Standard Database Operations Work in Ecto for Elixir

Introducing Our Example Phoenix App

Standard Inserts with Ecto

The Schema’s Role

The Repo’s Role

Creating a Record in the Database

The Performance Problem and Why Batch Operations Matter

Understanding `Ash.create` and `Ash.read`

Using `Repo.insert_all/3` To Bulk Insert Data

Handling Batch Updates With `Ecto.Multi`

An Example Using `Ecto.Multi` to Run a Batch Update

The Basics of `IO.inspect` for Elixir

Caveat: When `IO.inspect` Is the Last Call in a Function

Using `dbg/2` for Richer Inspection

When `IO.inspect` is Not Enough

Setting Default Options for `IO.inspect` in IEx

A Word of Caution About Elixir Packages `typed_struct` and `domo`