Erlang's Observer is often discussed in passing and regarded as a curiosity during Elixir courses. However, Observer provides many powerful tools for monitoring and debugging your application, both in development and production.

Together, we will learn how to access the Observer GUI and debug a project that leaks memory, both locally and through a remote node. We will set up process tracing and track garbage collections to find the offending code in our sample project.

Let's get started!

Set Up: Verifying Observer in Your Elixir Installation

Start by opening an IEx session:

iex

Now try to start the Observer GUI:

iex> :observer.start()

The return value :ok should appear and a new window should open.

If you get an error message about a missing :observer module, your Erlang installation might have been compiled without the :observer or :wx modules, or WxWidgets might be missing from your system. Refer to the instructions of your toolchain manager of choice to follow along.

The Observer GUI in Erlang for Elixir

The GUI has 9 separate tabs:

• System shows an overview of your system's capabilities, memory and CPU usage, and runtime statistics.

  

• Load Charts displays live memory, scheduler utilization, and I/O statistics.

  

• Memory Allocators shows the Erlang runtime's memory carriers usage.

• Applications shows the different running applications and their processes' supervision trees.

• Processes lists processes, the number of reductions, memory usage, and mailbox sizes.

• Ports and Sockets, respectively, list open Erlang ports and sockets and their owner processes.

• The Table Viewer tab lists viewable ETS tables. You can use the View > View Unreadable Tables OS menu to show more tables than the default state displays.

• Trace Overview, where we can set up tracing and view traces for offending processes, which we will use extensively in the next steps of this article.

At this point, if you have already used the Phoenix Framework and the LiveDashboard package, you might notice that there is some intersection between what Observer and LiveDashboard provide. However, LiveDashboard is more tailored to monitoring web applications.

I encourage you to check out these various tools before continuing. As an example, try selecting applications to see their supervision tree, or inspect the contents of ETS tables.

We will now proceed to set up a project with a faulty process and debug it using Observer.

Project Setup: Elixir App with Leaking Memory

We will create a simple Elixir application with a process that periodically leaks memory. This problem can occur in real applications if multiple processes get hold of large binaries, because the runtime shares large binaries stored in a dedicated heap instead of copying them.

Creating the Project

In your terminal, create the project with Mix, including a supervision tree, and change directory to this new folder:

mix new demo_app --sup
cd demo_app

Add the Offending Process Module

Create a new lib/demo_app/offending_process.ex file:

defmodule DemoApp.OffendingProcess do
  use GenServer
  require Logger
 
  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
 
  def init(_opts) do
    schedule_work()
    {:ok, %{binaries: []}}
  end
 
  def handle_info(:work, state) do
    large_binary = :crypto.strong_rand_bytes(512_000)
 
    new_state = %{state | binaries: [large_binary | state.binaries]}
 
    IO.puts("OffendingProcess created binary.")
 
    schedule_work()
 
    {:noreply, new_state}
  end
 
  defp schedule_work do
    Process.send_after(self(), :work, 2_000)
  end
end

Update the Application Supervisor

Edit lib/demo_app/application.ex:

defmodule DemoApp.Application do
  use Application
 
  @impl true
  def start(_type, _args) do
    children = [
      # Add the OffendingProcess to the supervision tree
      DemoApp.OffendingProcess
    ]
 
    opts = [strategy: :one_for_one, name: DemoApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Add the Observer Backend

Edit mix.exs to add the :runtime_tools application:

def application do
  [
    extra_applications: [:logger, :runtime_tools],
    mod: {DemoApp.Application, []}
  ]
end

Let's start the application in IEx with a named node, so we can connect to it from another node running Observer.

iex --name demo@127.0.0.1 --cookie democookie -S mix

You should see messages printed every 2 seconds.

OffendingProcess created binary.

Connecting Observer from a Separate Node

In a new terminal, start a second hidden IEx node.

iex --hidden --name observer@127.0.0.1 --cookie democookie

Then in the IEx session, connect to the first node and start Observer:

iex> Node.connect(:"demo@127.0.0.1")
true
iex> :observer.start(:"demo@127.0.0.1")
:ok

If you navigate to the Processes tab, you should see the OffendingProcess's memory usage slowly increase as the number of references to large binaries it holds grows.

In the System tab, you can see overall memory usage and binary-specific memory usage increase more quickly.

This two-node setup means we can avoid interfering with resource usage measurements on the first node, by not running the Observer app directly on it.

Debugging a Remote System with Observer

Let's set up tracing for our offending process from the Observer GUI controlled by our hidden node. Go to the Processes tab, right-click on Elixir.DemoApp.OffendingProcess, and select "Trace selected processes".

In the process options, select the following:

• Trace function call
• Trace arity instead of arguments
• Trace garbage collections

Move to the Trace Overview tab and click on the process now listed in the rightmost part of the GUI.

Click on the "Add Trace Pattern" button at the bottom of the GUI. In the pop-up that opens, select the module that we are looking to trace: Elixir.DemoApp.OffendingProcess.

In the second pop-up that opens, select functions to trace. We will select schedule_work/0.

The next pop-up asks us to select or write a Match Specification for our traces. If you are already familiar with :ets (the Erlang Term Storage application), the match specifications used to perform queries with :ets.select/2 are the same concept, despite having a slightly different grammar. Match specifications are detailed in the documentation of the Erlang Runtime System Application (ERTS).

For our purpose, we will select the pre-existing "Return Trace" match specification. Here, the return_trace function causes a return_from trace message to be sent when the selected function returns.

In the leftmost part of the window, click on "Start Trace".

A window will open where each schedule_work/0 call will log the entry and return from that function.

After a while, you will notice a different log: minor garbage collections triggered by the VM. In short, a garbage collection is an attempt to free memory by deleting data that cannot be referenced anymore.

In the screenshot, we can read information about the garbage collection that just occurred. All sizes are expressed in words.

• old_heap_block_size is the size of the memory block storing the heap pre-collection
• heap_block_size is the size of the memory block storing both the heap and the stack
• mbuf_size is the size of the process's message buffers
• recent_size is the size of the data that passed the previous garbage collection
• stack_size is the size of the stack
• old_heap_size is the size of block heap actually used, pre-collection
• heap_size is the size of block heap actually used
• bin_vheap_size is the size of binaries referenced from the process heap
• bin_vheap_block_size is the size of binaries allowed in the process's virtual heap before triggering a garbage collection

The key that interests us the most here is bin_vheap_size, describing the space taken by binaries referenced in the process. We are talking about a virtual heap here because large binaries aren't actually stored in the process's heap, but in a special and global binary heap as an optimization. In writing this article, when I first started tracing, this key read 7_168_092 words. After a while, it grew to 44_096_000 words.

Indeed, going back to the "System" tab in the Observer GUI, I can see 372MiB taken by the binaries.

And this concludes our overview of the Observer GUI!

Going Further

As you can see, the Observer GUI's capabilities are quite extensive and it is difficult to provide an exhaustive tour here.

Here are a few things you can try yourself to dive a bit deeper:

• Find the supervision tree of an application in the "Applications" tab, see its supervision strategies, and terminate a few of its children
• Inspect the state of a long-running process. Can you see all of its internal state in your IEx session?
• Trace specific functions through your application, then try to trace them without Observer, with :erlang.trace/3
• Try to start etop, included with the Observer application, with erl -name etop -hidden -s etop -s erlang halt -output text -node demo@127.0.0.1 -setcookie democookie and see how this compares to the graphical Observer
• Explore the trace tool builder ttb and crashdump_viewer, two other tools that come with the Observer application.

Wrapping Up

In this post, we took a quick tour of Erlang's Observer GUI. We first set up Observer for an Elixir application, then used an example app with a memory leak to explore some of Observer's capabilities.

We debugged a remote system, before finally suggesting some ways to dive deeper into exploring Observer's many functions.

Happy coding!