Writing tests is an essential part of any Elixir developer's routine. We're constantly chasing knowledge on how to write better tests, improving their speed and readability.

In this three-part series, we'll explore test data generation in Elixir. Whether you are a mid-level or senior-level Elixir developer, this series will provide valuable insights to help improve the testing process for your projects.

Let's start by digging into where test factories come from and why they are so popular in Elixir.

What are Test Factories in Elixir?

Test factories are functions for generating data, commonly used in the :test environment.

Test factory libraries like ExMachina are widely used among Elixir developers. If you look at the hex.pm stats, you'll see that ExMachina downloads aren't far behind the most popular Elixir database library: Ecto.

They allow you to create complex data structures using a very convenient interface that requires few inputs. Here's an example using ExMachina:

iex> user = ExMachina.insert(:user)
%User{
  id: "usr_xlkt"
  name: "Abilidebob"
  accounts: %Account{
    id: "acc_xktt"
    #....
  }
  #...
}

With a few characters, I can grab an example of a :user in the system. Test factories are inspired by the factory method pattern, a design pattern that allows a caller to create objects without knowing the specific module or class of the data that will be created.

Okay, this might sound a little bit complicated! Let's look at the following ExMachina code:

user = ExMachina.insert(:user)
 
do_something(user)

In this example, ExMachina is using the :user atom to dispatch to a function that can create examples of %User{} structs. If you write your tests in a dynamic style, you can rename the User struct without refactoring any tests, except the factory function.

The factory pattern isn't only helpful in tests. You can also use it to create or dispatch functions from dynamic sources, such as user inputs.

For example, you could use a factory function to create an account based on a user's input. A factory function is useful here because your application's users don't know the name of your source code's modules or functions — and you should keep it that way. You need to create a mechanism that links a user's input with the correct constructor:

def new_account("hobbyist"), do: HobbyAccount.new()
def new_account("professional"), do: ProfessionalAccount.new()
def new_account("enterprise"), do: EnterpriseAccount.new()
def new_account(account_type), do: raise ImpossibleAccount, account_type

In the previous example, I built a factory function named new_account/1 that takes a string as an argument and uses pattern matching to determine which function to call. For example, if a user selects the "Hobbyist" option from a selection box, the application will receive a "hobbyist" string as the input. The new_account/1 function can use the string to dispatch the HobbyAccount.new/1 function and generate a HobbyAcccount struct.

Why Use Test Factories for Elixir?

Test factories are good options for long-term test suite maintainability because of the following:

• Learnability - factories document what data structures can look like in production
• Reusability - data examples generated by factories can be reused in many different tests
• Productivity - developers can invoke complex data examples from factories by typing a few characters instead of building new examples from scratch every time
• Changeability - tests that rely on central factories always have the most up-to-date examples

These properties aren't inherently part of the factories alone. A software development team's discipline is required.

For example, if you don't provide relevant real-world examples in your factories, they will not be a source of knowledge for developers. If your factories produce values that other developers tend to override, they will more likely stop using those factories because they are not convenient anymore.

But when a factory pattern is well-maintained, it helps developers to maintain a sustainable test suite that lasts for a long time.

Now we'll turn our attention to test fixtures.

Test Fixtures in Elixir

In testing, a fixture is data that we prepare before running a test. Factories are functions that generate test data on demand.

Factories can complement fixtures — you don't necessarily have to opt for one over the other. For example, you can prepare a test fixture that uses a factory function to insert data in a database.

That's why we can sometimes use these terms interchangeably and still be understood. Often, people associate the term "fixtures" with the feature in Ruby on Rails. Ruby On Rails fixtures have the following properties:

• Data is defined in YAML files, not inline in the tests.
• All defined data is loaded in the database before each test run.
• You can reference these fixtures through dynamic methods generated by the framework.

However, different frameworks or libraries implement test fixtures differently. After all, the main purpose of test fixtures is to provide a common starting point for your tests.

There are basically two main kinds of test fixtures: inline and implicit.

Inline Test Fixtures

The simplest inline fixture you can do in Elixir is the following:

test "creates an author profile" do
  user = MyApp.Repo.insert!(%User{id: "usr_123", name: "Abilidebob"})
 
  # rest of the test code
end

With a basic Repo.insert!/1, we make a database fixture for a test. That test can rely on the user with the id "usr_123" always being present in the database.

This example code assumes the test suite uses Ecto sandbox setup, which always reverts database changes made during a test after its execution.

So we don't need to worry at the beginning of the test if the user with the same id is already inserted in a previous execution. We can even use a factory function to insert the data, and it still would be correct to call it a test fixture. This fixture is explicit and inline with the tests.

Implicit Test Fixtures

Like in Ruby on Rails, it's possible to create implicit fixtures that multiple tests can share using the ExUnit.CaseTemplate test template:

# test/support/data_case.ex
defmodule MyApp.DataCase do
  use ExUnit.CaseTemplate
 
  # ... bunch of other code here
 
  setup _context do
    user = %User{id: "usr_123", name: "Abilidebob"}
    %{user: MyApp.Repo.insert!(user)}
  end
end

The code above uses the setup directive to make all tests that use this template insert a user. Then we return a map, where the key is :user, and the value is our recently created user. Any test can quickly reference this by using the key :user and grabbing it from the test context. For example:

defmodule MyApp.MyTest do
  use MyApp.DataCase
 
  test "hello world", %{user: user} do
    # then I use my `user` here
  end
end

ExUnit Contexts

ExUnit test contexts are a flexible mechanism where you can build things to share between multiple tests.

The biggest disadvantage of fixtures is their impact on test suite speed and test isolation if you share them too much. The speed can decrease because all the code you put in setup blocks of test templates always runs before each test, even when you don't need the data from the test context.

So be careful how much data you add and how multiple modules use the templates. Otherwise, as the speed decreases, implicit coupling between multiple tests on the same test fixtures can also increase. A small change to a test fixture shared in this way can make multiple tests fail for no obvious reason.

Sharing Bypass Instances Between Multiple Tests

Developers favor factory libraries over Ruby On Rails fixtures because they'd rather have testing data explicitly invoked from tests than implicit data generated far from where it is needed. In other words, they prefer inline test fixtures.

Explicitness contributes to better long-term maintenance. Implicit test fixtures are better for things that are very cheap to build and have a very general purpose, and low coupling with testing specifics.

A good example is the Bypass instance:

# test/support/data_case.ex
defmodule MyApp.GithubIntegrationCase do
  use ExUnit.CaseTemplate
 
  # ... bunch of other code here
 
  setup _context do
    bypass = Bypass.open()
    client = GitHub.new(endpoint: "http://localhost:#{bypass.port}/")
    %{bypass: bypass, client: client}
  end
end

In the example above, I prepare a test template for my GitHub client integration tests. The setup code creates a fake HTTP server using Bypass. The GitHub client uses this server to make HTTP requests during the tests using the client settings.

Now, all tests that use MyApp.GithubIntegrationCase can use the :bypass and :client keys to pull the fake server and its settings, respectively, from the test context.

Here is how these fixtures might be used in a test:

defmodule MyApp.MyTest do
  use MyApp.GithubIntegrationCase
 
  test "user registration", %{bypass: bypass, client: client} do
    Bypass.expect(bypass, :post, "/users") do
      # ... setup expectations and return a success response
    end
 
    # Use the `client` tag to make HTTP requests with the `GitHub` client
    {:ok, user} = GitHub.create_user(client, name: "Abilidebob")
 
    # Assert that the user is correct
    # ...
  end
end

If you've worked with other frameworks, such as Ruby on Rails, you might be surprised that Elixir doesn't have a popular fixture library.

I believe that's because the ExUnit test context can share test fixtures pretty easily. The fixture pattern is usually discouraged due to its impact on test suite speed and maintainability.

Test Factories and Your Elixir App's Rules

In his excellent post Towards Maintainable Elixir: Testing, Saša Jurić provides valuable tips on how to maintain a test suite. The focus of the post is on increasing your confidence in tests and reducing test overlap. There is no mention of test factories. Instead, Saša uses his own application interface to prepare test data. The reason he avoids test factories is because of their biggest problem: they bypass your application's rules.

When you set up a test factory for your test suite, the data examples it generates are completely detached from your application code's rules.

For example, let's say you have a User entity with an is_author flag, but that flag is only set to true if the user has one author profile. Here's an example of how the production code would look:

def create_author_profile(user) do
  author_profile = build_author_profile(user)
  user = Ecto.Changeset.change(user, %{is_author: true})
 
  Multi.new()
  |> Multi.update(:user, user)
  |> Multi.insert(:author_profile, author_profile)
  |> Repo.transaction()
end

Let's say that you now want to create the :user and :author factories for your test factory.

Here's an example using the ExMachina library:

def user_factory do
  %User{name: "Abili"}
end
 
def author_factory do
  %Author{
    name: "Mr. DeBob",
    user: build(:user, is_author: true)
  }
end

In the example above, the factories are straightforward. They build structs and assign values that make sense. However, have you noticed the is_author: true part? That tiny part is actually the biggest problem with factories: we have to duplicate our business rules!

Imagine having to remember to replicate every single aspect of multiple data modifications in your factories. Not only does this add extra code that needs to be maintained, but it can also mislead developers into trusting invalid data that your app cannot produce.

The invalid data generated by factories can lead you to write unnecessarily defensive code. For example, if someone forgets to include is_author: true in the factory, you might write code like this:

if user.is_author || Profiles.find_author(user_id) do
# logic for users that are author here

In the previous example, a check is written for the is_author flag. It is assumed the flag isn't always reliable — after all, the factory generates authors with is_author: false users. As a result, we decide to make a query in the database to double-check if the user is an author.

As you can see, factories that are decoupled from application rules aren't always perfect and can cause problems if they are not properly maintained.

Despite their issues, factories are a popular pattern for generating test data in Elixir. They offer convenient functions to build complex data structures, and you can invoke them on demand.

However, using factories adds an extra layer of maintainability, an extra worry to put in your team's minds. The worry is usually negligible when your application code is small, but can become problematic as your business and code grow more complex.

Up Next: Generating Data Functions in Your Elixir App

In this first part of our three-part series, we introduced test factories and test fixtures for Elixir. We then explored the biggest issue with test factories: the fact that they bypass your application's rules.

In part two, we'll explore how you can combat this by creating data generation functions using your application's rules.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!