Over time, web applications have evolved from simple request/response-based systems into complex, distributed ones with lots of moving parts. If something goes wrong (and you can be sure it will), finding the cause can be nearly impossible. But this need not be the case: enter tracing.

Tracing refers to the process of collecting detailed information about the execution of requests within an application, including function calls, execution time, and other relevant data.

In this article, we'll learn what tracing is all about, including the benefits and challenges of tracing. In addition to this, we'll look at distributed tracing and show you how to set up tracing using AppSignal for a Phoenix LiveView financial news tracker app.

Prerequisites

To follow along with this tutorial, you'll need:

• Elixir and the Phoenix framework installed. You can follow this guide to set them up.
• An Elixir application. For this tutorial, we'll use the Phoenix LiveView financial news tracker app.
• An AppSignal account. If you don't have one, you can sign up for a free trial.
• Some experience working with Elixir and the Phoenix framework.

With that done, let's learn what tracing is all about.

Understanding Tracing

Tracing is the process of collecting data during the runtime execution of an application. It gives you visibility into what an individual service (an internal module or function, an external integration, or a combination of all these) does as part of a request cycle. Essentially, the term "tracing" includes both "application tracing" and "distributed tracing".

Application Tracing vs. Distributed Tracing

Application tracing and distributed tracing can be collectively termed as "tracing". However, with application tracing, you follow the execution path of a request within an application or a single service. For example, tracing a user authentication flow that starts and completes within an application without calls to an external system is a good example of an application trace.

But now imagine a distributed system where an app has to interact with external services. The news tracker application which we'll use throughout this article is a perfect example of this.

Here, a request crosses several service boundaries — from the app, through one or more background-job services, to the news-fetching API, then into a cache service that stores the fetched news items, and finally back to the user's browser. Even though this example is simple, tracing such a request flow goes beyond what happens inside the app; it covers every service and system the request touches. This is what we mean by "distributed tracing".

Why We Trace

Without tracing in place, it's difficult to understand what is going on with your app, which ultimately makes it challenging to measure the impact that various issues have on the application.

Here are some reasons why you might want to trace:

• Performance analysis - There are many factors that could affect the performance of an Elixir application, such as slow API calls, unoptimized database queries, network latency, and more. But if you have tracing in place, you can pinpoint the issues slowing down your app and fix them.
• Helping with debugging - Using the example news tracker app, let's say users submit tickets about not getting news updates on time. Is the problem the API, the database, the news cache, or a combination of all three? Without proper tracing, finding where the issues are coming from can become a real headache for app developers.
• Auditing - From a compliance standpoint, tracing can be used to check if your application is leaking any sensitive data (such as usernames or emails) in server logs and other places where this data should not be.

Of course, there are other reasons why we trace, but we'll leave it at that for now. Next, let's see why you should use AppSignal for tracing your Elixir application.

Why Use AppSignal for Elixir?

AppSignal is an excellent application performance management tool with integrations for most modern languages and frameworks. AppSignal offers out-of-the-box integration for pure Elixir and Phoenix apps; for other frameworks and packages, you can add integrations using custom instrumentation, which isn't that difficult to set up.

The Elixir App

The application we'll work with throughout this tutorial is a Phoenix LiveView app that showcases a real-world application. It's a financial news tracker app that:

• Fetches financial news from the AlphaVantage API
• Manages user-specific news topic preferences
• Delivers real-time updates via LiveView
• Uses background jobs to fetch data
• Implements a caching layer to improve performance

You can see the architecture of the app below:

This architecture is perfect for demonstrating both application and distributed tracing.

In case you haven't done so already, go ahead and fork the app from this Github repository.

Installing the AppSignal Package

Open up mix.exs and add the AppSignal package as shown below:

# mix.exs
...
defp deps do
    [
      ...
      {:appsignal_phoenix, "~> 2.13"}
    ]
end
...

After that, run mix deps.get to add the package to the application, then run the command below to install it:

mix appsignal.install <YOUR API KEY HERE>

After running the installer, you should be prompted for the name of the app:

Validating Push API key: Valid! 🎉
What is your application's name? [email_subscription_app]: Email Subscription App

After that, you'll need to choose the configuration method:

There are two methods of configuring AppSignal in your application.
  Option 1: Using a "config/appsignal.exs" file. (1)
  Option 2: Using system environment variables.  (2)

Choose the option that suits you best, although the config file option makes it easier to customize the configuration as you wish.

Next, since we are working with Phoenix LiveView, we need to take some extra setup steps.

Adding AppSignal Phoenix Support

Support for Phoenix instrumentation comes in a separate package that depends on the main package we just installed. To add the Phoenix support package, add it to mix.exs:

defmodule FinanceNews.MixProject do
  ...
  defp deps do
    [
      ...
      {:appsignal, "~> 2.8"},
      {:appsignal_phoenix, "~> 2.0"}
    ]
  end
end

Then add the LiveView telemetry integration in the app's application.ex file, like so:

defmodule FinanceNews.Application do
 
  use Application
 
  def start(_type, _args) do
    Appsignal.Phoenix.LiveView.attach() # add this line
    children = [
      ...
    ]
    ...
  end
end

Tip: AppSignal offers two different ways to instrument live views, using an automatic handler or through manual instrumentation.

And with that, we have what we need to start tracing our app's requests. But, before we move on to the actual implementation, it's very important we understand the building blocks of tracing and the types of data we want to collect using our tracing setup.

The Building Blocks of Tracing

As we've already mentioned, tracing is about understanding the journey a request takes through a system. That said, a trace is fundamentally made up of several building blocks such as transactions and spans, trace context, sampling, and tags and attributes.

Transactions and Spans

A "transaction" represents a complete unit of work in the application. You can visualize this using the structure of a tree, where the root is the initial request and the branches (spans) are the individual operations that make up that request.

A span contains information like the name of the operation, the start and end times of the operation, a list of events that took place during the operation, and more.

Trace Context

This is a very important concept in distributed tracing, since it defines the relationship between different parts of a transaction across different services and even across system boundaries. For example, and as you will soon see, when our news tracker app makes an API call to fetch the latest news updates, it's a good idea to know what is going on before, during, and after the API call. This is what we mean by "trace context".

The diagram below illustrates this concept using our news tracker app:

Sampling

As much as we want visibility into every part of an application, it would be very costly and probably even negatively impact your app's performance. At some point, you'll have to decide what to trace and what to ignore. This becomes especially important in distributed systems where requests can go through complex transactions and cross many service boundaries. Sampling is the art of choosing representative transactions that will provide the most value to the tracing activity.

Tags and Attributes

Tags and attributes are descriptive pieces of information you use to add context to your traces.

What to Trace vs. What to Ignore

When deciding where to set up traces in your application, understanding what data to collect can be the difference between an effective and ineffective trace implementation.

With that in mind, here are some valuable types of tracing data:

• Request/response data - In the example app we're using, having visibility into how various requests and responses are happening is very valuable.
• Performance metrics - For example, it would be good to know how long a LiveView render of the latest news articles takes.
• Errors - If an error occurs (let's say when fetching news items from the API), it is valuable to know how the error started and progressed.
• Custom data - One of the benefits of using a tool like AppSignal is the ability to use custom instrumentation to capture data that's unique to your application.

Now that we know what data we should collect using a tracing setup, let's implement it, starting with basic tracing, before moving on to more advanced tracing methods later.

Implementing Tracing

To get started with tracing, we need to:

• Set up a data capture layer using instrumentation.
• Send the tracing data to a third-party visualization system.
• Analyze the data for further action.

Since we are using the AppSignal library for Phoenix/Elixir, we get the first two without much effort on our part. The last bit is on us completely.

Tracing Web Transactions

In a Phoenix application, web transactions are the entry point of user interactions with our application, and will often be the trigger for subsequent and more advanced transactions later on.

Using our example application, we can break down a web transaction as shown below:

• Initial request for information
• Controller and LiveView actions
• Database queries
• Rendering views

To see how these are represented on the AppSignal dashboard, consider the handle_event/3 "save_topics" function below:

# lib/finance_news_web/live/topic_live.ex
 
defmodule FinanceNewsWeb.TopicLive do
  ...
 
  def handle_event("save_topics", _, socket) do
    user = socket.assigns.current_user
    topics = MapSet.to_list(socket.assigns.selected_topics)
 
    case Topics.update_user_topics(user, topics) do
      {:ok, _} ->
        {:noreply, push_navigate(socket, to: ~p"/feed")}
 
      {:error, _} ->
        {:noreply, assign(socket, error_message: "Failed to save topics. Please try again.")}
    end
  end
end

From this code snippet, you can see:

• An interaction with the database to fetch the currently logged-in user
• Another interaction with the database to fetch or update the user's selected topics
• Rendering the FeedLive LiveView or displaying an error message in case something goes wrong

A request that makes use of this function will be traced and represented on your dashboard in AppSignal, as shown below:

The trace shows:

• The name of the event
• A breakdown of the sample showing the time taken by the request while interacting with different services in the app. In this case, the Ecto transaction takes the most amount of time compared to the LiveView render.

But what does this really tell us? Or, put another way, how does this information help us?

Analyzing the Results

One use case for such trace data is to help you establish your app's baseline performance. What this means is that, by collecting such data over time, you will soon establish what is acceptable versus what is not in terms of performance.

For example, over time, you might discover topic updates take on average 20-30ms to load, so if you come across trace samples that show loading times of 50ms+, you would know something is amiss. You can then put performance alerts in place, with a setup that uses AppSignal's custom instrumentation, as below:

# lib/finance_news_web/live/topic_live.ex
defmodule FinanceNewsWeb.TopicLive do
  ...
  def handle_event("save_topics", _, socket) do
    Appsignal.instrument("Topics.save_topics", fn ->
      user = socket.assigns.current_user
      topics = MapSet.to_list(socket.assigns.selected_topics)
 
      case Topics.update_user_topics(user, topics) do
        {:ok, _} ->
          {:noreply, push_navigate(socket, to: ~p"/feed")}
 
        {:error, _} ->
          {:noreply, assign(socket, error_message: "Failed to save topics. Please try again.")}
      end
    end)
  end
  ...
end

Next, let's look at a more advanced example where we trace an external service.

Tracing External Services

You'll often find external services to be the most unpredictable part of any application. Such services can be integral to your app's functioning, but since you don't have 100% control, you'll find that they can fail, time out, or become slow without warning.

You can see that sourcing the latest finance news items is key to the functioning of our financial news tracker app. Yet this part of the app completely relies on an external service, the Alpha Vantage API.

As the API is important to our app's functioning, let's see how we can implement tracing for such an external service.

Note: AppSignal offers automatic instrumentation for HTTP libraries like Finch (we're using Finch in the news tracker app), so there's no need to add any instrumentation for now. But if you need a custom, more detailed trace, go ahead and use a custom instrumentation.

The results of tracing external service requests are shown below:

Details of the trace are also shown below (note how AppSignal conveniently places such traces under the appropriate "background" namespace):

And that's it!

Wrapping Up

In this article, we learned that tracing goes far beyond simple error tracking. By using various examples related to our financial news tracker application, we've seen how proper instrumentation provides complete request visibility, performance insights, and even deep information on errors.

With tracing in place, you'll have much greater visibility into how requests flow through your system, empowering you to see bottlenecks and optimize the performance and reliability of your Elixir apps. Combine this with the intuitive dashboards provided by AppSignal, and you have everything you need to professionally run an Elixir application in production.

Happy tracing!