Background task processing is something that many developers may encounter when building Elixir applications. This might include sending emails asynchronously, posting and fetching data from an API, and more.

Oban, a powerful and persistent job processing library, offers a reliable way to handle background tasks, scheduled operations, and more. However, like any complex system, Oban requires careful monitoring to ensure its smooth operation, identify bottlenecks, and prevent unexpected failures.

In this article, we'll learn about the fundamentals of Oban, its potential pitfalls, and how to leverage AppSignal's monitoring to gain valuable insights into your Oban setup.

Prerequisites

• Elixir and the Phoenix framework installed on your local computer.
• An AppSignal account. If you don't have one, sign up for a free trial.
• An Elixir/Phoenix app to play around with. If you don't have one, you can fork this one that we'll use throughout this tutorial.

Introducing Oban

Oban is a robust background job processing library for Elixir that uses PostgreSQL, MySQL, or SQLite 3 for storing job queues. Unlike other job processing solutions, Oban doesn't require Redis or other external dependencies beyond your existing database.

Key Features of Oban

• Persistence: Jobs are stored in the database, ensuring they survive application restarts.
• Concurrency Control: Jobs can be organized into queues with configurable concurrency limits.
• Scheduled Jobs: It offers support for one-off and recurring jobs with precision scheduling.
• Job Prioritization: Within each queue, jobs can be assigned different priority levels.
• Error Handling: Built-in retry mechanisms.
• Unique Jobs: Oban provides mechanisms to prevent job duplication with uniqueness constraints.

In terms of structure, Oban consists of workers, which are the modules that define job processing logic, queues, and supervisors, used to manage job execution. This architecture makes it very good for reliable background processing in Elixir apps.

Introducing Our Example App and Setting up Oban

We'll use a simple email subscription app for this tutorial. A user will be able to submit their email address, and the app will then send a scheduled email using background job queues powered by Oban.

This app should provide enough context for you to learn about Oban and how to monitor it using AppSignal.

Adding Oban to an Elixir Project

Oban's installation is very straightforward. Begin by adding the following packages to your app's mix.exs file:

# mix.exs
...
 
{:oban, "~> 2.19"},
{:igniter, "~> 0.5", only: [:dev]},
...

Next, fetch the newly added dependencies with mix deps.get.

Installing Oban Using Igniter

Previously, after adding Oban and running the mix oban.install command, we had to modify the generated migration file manually.

But now, with the addition of the Igniter library, after running mix oban.install, the manual modification of the generated migration file is not needed.

Now that you know what Igniter is for, go ahead and install Oban with:

mix oban.install

Then run the generated migrations with mix ecto.migrate.

Adding AppSignal to the Project

Open up the mix.exs file, then edit it as shown below:

# mix.exs
...
defp deps do
    [
        ...
        {:appsignal, "~> 2.8"},
        {:appsignal_phoenix, "~> 2.0"}
    ]
end
...

Note: The AppSignal instrumentation for Phoenix apps is available through a separate appsignal_phoenix package, which depends on the core Elixir appsignal package.

Then run mix deps.get to fetch the package, followed by the command mix appsignal.install <YOUR APPSIGNAL API KEY> to install AppSignal.

You can choose how you want AppSignal configured for the project from the following options:

• Customizing the name of your application.
• Choosing the configuration method to be used (two options are available: via an environment variable or through the use of a configuration file).

Once you've made your choices, you should get an output similar to the one shown below if the process runs without issues.

Validating Push API key: Valid! 🎉
What is your application's name? [email_subscription_app]: email_subscription_app
There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)
What is your preferred configuration method? [1]: 1
Writing config file config/appsignal.exs: Success!
Linking config to config/config.exs: Success!
Activating dev environment: Success!
Activating prod environment: Success!
...
AppSignal installed! 🎉

Now you should have AppSignal installed, and your app will start sending data to your dashboard:

So far, we have a Phoenix application integrated with Oban and AppSignal. We can now turn our attention to the question of how to monitor Oban.

But before we do that, let's learn about some of Oban's challenges and potential points of failure for insights into where to focus our monitoring efforts.

Common Oban Challenges and Potential Points of Failure

Like any background job framework, Oban can encounter various challenges that affect performance and reliability. Understanding these potential issues is crucial for effective monitoring and maintenance. Let's explore some of these:

• Queue overloading - When jobs are enqueued faster than they can be processed, queues can build up rapidly. This backlog can lead to increased memory usage, slower overall system performance, and delayed job execution. It is particularly problematic for time-sensitive operations.
• Failed jobs and error handling - Jobs can fail for various reasons, including database timeouts, API rate limits, and so forth. Without proper error handling and monitoring, failed jobs might slip through the cracks, or worse, create a sequence of other failures.
• Database contention - Since Oban uses your database for job storage, high job throughput can lead to database contention. This manifests as lock timeouts and increased query times and can impact your application's core functionality.
• Worker crashes - When workers crash unexpectedly, jobs might be abandoned or stuck in a "running" state, and without proper supervision, these orphaned jobs can clog your system.

While these examples represent common challenges with Oban, they're just a few of the potential issues you might encounter.

With this knowledge of Oban's architecture and potential pitfalls, we're now ready to explore how AppSignal can help us gain visibility into our job processing system and detect problems before they impact users.

Monitoring Oban Using AppSignal

Once you've added the AppSignal package, it will automatically instrument Oban job workers and queue metrics out of the box.

Tip: AppSignal also adds an Oban namespace, which you can use to filter for Oban-specific metrics.

A few Oban metrics are automatically made available. Let's start by looking at job execution time.

Job Execution Time Metrics

This metric captures the duration spent by a worker to perform a particular job. For example, in the email subscription app, when a user submits an email, the WelcomeEmailWorker is used to send a welcome email.

The code below shows the Oban worker responsible for processing the welcome email:

# lib/email_subscription_app/workers/welcome_email_worker.ex
 
defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}
 
  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()
    |> case do
      {:ok, _} -> :ok
      {:error, reason} ->
        raise "Mailer delivery error: #{inspect(reason)}"
    end
  end
end

And this is automatically captured on the AppSignal dashboard, as you can see below:

Metrics for Slow Jobs

AppSignal will also automatically capture slow jobs. The dashboard below shows you what this looks like:

AppSignal also automatically instruments Oban errors. Let's see how that looks next.

Automatic Metrics for Oban Errors

AppSignal provides excellent error instrumentation for Oban, as you can see from the screenshots below:

More details are made available when you click into a particular error:

While it's great having automatic instrumentation for some of the more common metrics, there will be times when you need to go beyond these.

Let's say you wanted to know the total number of welcome emails sent within a certain period of time or the number of failed ecommerce orders; such metrics might not be automatically captured. For custom metrics like these, it's necessary to implement custom instrumentation for Oban.

Custom Metrics for Deeper Insights

Using the email subscription app as an example, let's say we are interested in counting the number of successful email subscriptions that are initiated on the app. This is a rather simple metric, but it will give you an idea of what's possible with AppSignal.

To accomplish our task, we can add custom instrumentation to the WelcomeEmailWorker since this Oban worker is responsible for sending the initial user email.

Edit the worker's code as shown below:

# lib/email_subscription_app/workers/welcome_email_worker.ex
 
defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}
 
  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    Appsignal.instrument("welcome_email_worker", fn ->
      email
      |> EmailSender.welcome_email()
      |> Mailer.deliver()
      |> case do
        {:ok, _} ->
          Appsignal.increment_counter("successful_email.subscriptions", 1)
        {:error, reason} ->
          raise "Mailer delivery error: #{inspect(reason)}"
      end
    end)
  end
end

Here's what's going on:

• We use Appsignal.instrument/2 to wrap the worker's perform function and create a span (which will be sent over to AppSignal as an event).
• Then, we use Appsignal.increment_counter/3 to count the number of successful email deliveries, which we use as the sign of a successful subscription.

Once that is done, we need to set up a custom dashboard to visualize this metric.

Setting up Custom Dashboards for Oban on AppSignal

Start by clicking on the "Add dashboard" button on the left-hand side menu:

Then the "Create a dashboard" button on the next screen:

Now give the new dashboard a title and description:

Adding a Custom Oban Graph

Next, click on the "Add graph" button of your new dashboard:

Then you'll see the fields needed for the custom graph:

Here is a brief outline of what you're looking at in terms of the fields:

• a. Graph title - Input a relevant title for the graph.
• b. Graph description - Add a description.
• c and d. Graph metrics - This one needs a bit more of an explanation. Metric here is the metric we added to our app's counter function. In the email subscription app, this is successful_email.subscriptions (as you can see from the metrics in the screenshot below):

Note: I won't go into the details of all the custom graph and dashboard features available, but you can explore them further in the dashboards documentation.

Finally, when you add a custom graph, you'll start seeing the custom metrics:

And that's it!

Wrapping Up

In this article, we've explored how to integrate and monitor Oban with AppSignal in an Elixir/Phoenix application. We started by setting up Oban for background job processing and then added AppSignal for monitoring.

We discussed common challenges with Oban and how AppSignal's automatic instrumentation can help you gain insights into job execution times, slow jobs, and errors. Additionally, we demonstrated how to create custom metrics and dashboards for more granular monitoring tailored to your application's needs.

By leveraging these tools, you can ensure your background job processing is efficient, reliable, and well-monitored, ultimately leading to a more robust and performant application.

Happy coding!