When my Sidekiq job starts failing or slowing down, I often feel frustrated, especially if I don’t know how to fix it.

If you’re using Sidekiq to run your background jobs, you know what I’m talking about. It’s a vital element of your stack, handling everything from data exports to password reset requests. It runs silently in the background, and most of the time, you’re not even giving it a second thought.

However, when things go wrong, like pages loading slowly, emails never being sent, or exports failing, the impact can be huge. Getting to the bottom of the issue can be like finding a needle in a haystack.

That’s where AppSignal comes in. It can monitor your Sidekiq jobs, alert you when things go wrong, and give you all the context you need to fix and rerun failed jobs without having to spend hours debugging or dealing with customer complaints.

In this tutorial, you will learn how to monitor your Sidekiq jobs’ performance with AppSignal. To follow along, you need to already have a Ruby application with some Sidekiq jobs.

Note: Even though we’ll be focusing on Sidekiq jobs here, it’s worth noting that AppSignal can monitor all aspects of your Rails application.

The Problem

Sidekiq jobs live in a different part of the web application, which can make issues hard to spot.

When a controller action fails, you get an alert within seconds. But what happens when a background job silently retries four times and gives up? You usually find out only when a user emails support asking where their CSV export went. These jobs slowly start piling up in the queue and degrading in performance without any obvious signals.

When Sidekiq jobs fail, you might see things like:

• Queues backing up and throughput collapsing
• Vital jobs silently exhausting their retries and dying
• Exceptions getting swallowed with no trace in your logs
• Users feeling the impact long before you realize something is happening

What You Get Out of the Box

A great thing about AppSignal is how easy it is to get started with. You don’t need a bunch of customization. Just add the appsignal gem, and you’re good to go.

From that point on, every Sidekiq job execution is tracked automatically. You get insights like job duration, queue wait time, and many other instrumented events that fire within the job itself. In practice, this means you get a structured performance data and error visibility of your jobs immediately after AppSignal has been activated.

If you’d like a full breakdown of what is captured and how the integration works under the hood, the AppSignal Sidekiq documentation is worth a look. However, here’s a categorized list of what you get out of the box:

• Job duration and throughput per worker, or how long each job takes and how many are being completed;
• Queue length and latency, or how many jobs are waiting and how long they've been sitting there;
• Job status per queue, which is a breakdown of succeeded, failed, and retried jobs across every queue;
• Job count, which is the number of workers and processes that are active at any given time;
• Connection count, or the number of open Sidekiq connections;
• Redis memory usage, which refers to both allocated and RSS memory consumed by your Redis instance.

Three Signals That Matter

Instead of just going through different random job performance metrics, we will focus on the three most important ones:

1. Job duration lets you spot your slowest jobs before they become outages. A job that averaged 2 seconds and now averages 12 is a problem you want to catch early.
2. Queue latency has to do with the time a job spends sitting in the queue waiting to be picked up by a worker.
3. Error rate and retry count provides a comparison between the error rate and the retry count.

You will learn how to find those metrics in the AppSignal dashboard and why they actually matter. But, before we go ahead with this section, it’s worth mentioning that Sidekiq provides its own dashboard, as well.

Still, as you can see from the image below, this dashboard doesn’t give you the full picture. It misses some important metrics and additional insights that are available on the AppSignal dashboard.

Adding AppSignal to the Ruby/Sidekiq Project

Assuming you already have a project set up (as stated in the introduction), these steps will guide you through how you can add AppSignal to your Ruby project. And since AppSignal automatically tracks the Sidekiq metrics, there isn’t much configuration necessary on your end.

Start by adding 'appsignal' to your Gemfile, then:

bundle install

Next, sign in to AppSignal and follow the steps below:

• Create a new application using the “Add app” button.
• Choose Ruby & Rails as the language.
• Fill in the app name.
• Install Rails for your framework.
• Run the CLI installer provided in the root of your project and follow the instructions. The installer should look like this bundle exec appsignal install YOUR_PUSH_API_KEY .

This creates config/appsignal.yml automatically. Here’s what mine looks like:

# AppSignal Ruby gem configuration
# Visit our documentation for a list of all available configuration options.
# https://docs.appsignal.com/ruby/configuration/options.html
Appsignal.configure do |config|
  config.activate_if_environment("development", "production", "staging")
  config.name = "Sidekiq"
  # The application's Push API key
  # We recommend removing this line and setting this option with the
  # APPSIGNAL_PUSH_API_KEY environment variable instead.
  # https://docs.appsignal.com/ruby/configuration/options.html#option-push_api_key
  config.push_api_key = "APPSIGNAL_PUSH_API_KEY"
 
  # Configure actions that should not be monitored by AppSignal.
  # For more information see our docs:
  # https://docs.appsignal.com/ruby/configuration/ignore-actions.html
  # config.ignore_actions << "ApplicationController#isup"
 
  # Configure errors that should not be recorded by AppSignal.
  # For more information see our docs:
  # https://docs.appsignal.com/ruby/configuration/ignore-errors.html
  # config.ignore_errors << "MyCustomError"
end
 

Update the config/boot.rb (the file that contains the setup for processing jobs) to load AppSignal before Sidekiq:

require 'appsignal'
require 'appsignal/integrations/sidekiq'
 
Appsignal.start
 
require 'sidekiq'
require 'app/jobs/welcome_email_job'
require 'app/jobs/order_processing_job'
require 'app/jobs/log_cleanup_job'
 
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
end
 
Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
end
 

Now you can run Sidekiq as normal:

bundle exec sidekiq -r ./config/boot.rb -C ./config/sidekiq.yml

AppSignal will automatically track overall job status, job status per queue, duration per worker, and much more, which is enough for what we want to accomplish in this article.

Job Duration

A job that used to take 2 seconds and now averages 12 is your next outage waiting to happen. Any time a job starts taking longer than expected to execute, it’s a sign that something’s off.

For example:

• A database query inside the job is hitting an unindexed table as data grows;
• An external API the job depends on has slowed down or started timing out;
• A job that once processed 10 records is now processing 10,000 with no change to the underlying code.

You want to catch this early because that way, you’ll be fixing a performance issue instead of fighting a full-blown incident. You can easily spot this metric on Duration per worker graph on the dashboard, which shows the amount of time that it took jobs to execute, grouped by namespace and action.

Queue Latency

Latency is the time between when a job is enqueued and when a worker actually picks it up. It shows you how long a job has been sitting around before it starts.

This makes queue latency your earliest warning for backlog problems. It starts climbing before jobs begin failing or erroring, which gives you a chance to catch issues early. If you spot a spike on the AppSignal Sidekiq dashboard, it means your workers can't keep up with the current load, even if every job that runs is technically succeeding.

In the image below, you can see that there is no latency. However, if you have an application used by multiple users at a time, you’ll probably see some readings in the Queue latency chart.

Error Rate vs. Retry Count

Sidekiq retries can easily mask failures. A job that errors five times before actually succeeding looks fine from the outside. That’s where AppSignal provides a clearer picture by showing the actual error count and backtraces.

Without that visibility, you can end up missing real issues, such as:

• a payment processing job that retries silently after hitting a rate limit and fails repeatedly, which can cause the charge to never go through
• a CSV export job that fails three times due to a sudden memory spike before it finally succeeds, adding delays that the user has to deal with
• a SendWebhookJob to a partner API that fails three times with a 503 error before it gets a 200 response. Sidekiq marks it as successful; meanwhile, your partner is experiencing delays in data, and their support team is filing tickets with yours. Luckily, AppSignal shows the 503 errors and identifies which partner endpoint is having issues.

Sidekiq's built-in retry mechanism is great for resilience, but it can be a hindrance when monitoring your application because it can lead to you missing a potential problem. The job we’ve mentioned above that errors 5 times before eventually succeeding still shows up as zero failures in the Sidekiq web UI, and it just disappears from the queue as "done." AppSignal captures every failed attempt, its backtrace, and the exact job arguments that triggered it. Without this, you can't really know for sure how reliable your jobs are.

You can find these errors in the Dashboard overview.

Get Alerted of Failures Before Users Do

You don’t want to be sitting there, watching the metrics and searching for issues manually. That simply wouldn’t be productive. This is exactly where the Anomaly detection feature comes in.

It lets you set triggers that send notifications when a metric crosses a defined threshold. In the context of Sidekiq jobs, it means getting an alert the moment queue latency spikes or a job class starts returning errors at an unusual rate, ideally before any user opens a support ticket.

The detection runs on a minute-by-minute basis, so the gap between a problem starting and you knowing about it is kept to a minimum.

You can configure Triggers with a variety of metrics, including error count, throughput, error rate increases, slow actions, and queue time.

For the sake of this tutorial, we will set up triggers on the last three.

Before you do that, though, you need to create triggers, the functionality that executes these alerts.

Go to your app navigation and click on the Anomaly detection tab. Choose Triggers, then click the button to add a new trigger. Use the image above as a guide.

Slow Actions

The slow action trigger fires when the mean duration of a specific job class exceeds a time limit you’ve defined. With it, you can catch performance regressions, like when a job that used to run in 2 seconds suddenly starts running in 12 for one reason or another.

To set this up, go to Slow actions tab under Actions. Choose the following options and save the trigger:

• Namespace: background
• Alert me when the: Mean
• Is above (ms): 2000
• Tick Default email notifier

When the mean duration of that job crosses 3 seconds, AppSignal sends an alert with the job class name, the current mean, and a direct link to its performance page.

Error Rate Increase

The error rate trigger is activated when the percentage of job executions that raise exceptions goes above your defined limit. Without this kind of metric, a job could start failing 20% of the time and still go unnoticed for hours because Sidekiq retries keep the queue visually clean.

To set this up, click Error rate under Actions. Then select the following options, and save the trigger:

• Namespace: Background
• Alert me when error rate is above: more than, 5%
• Check Default email notifier

With the 5% error rate threshold set here, you get alerted after roughly 1 in 20 job executions fail.

Note: You will need to set a lower threshold for payment processing or sensitive jobs where even a 1% failure rate is unacceptable.

Queue Time

The queue time trigger fires when the latency between enqueuing a job and it being executed exceeds your threshold. This is the earliest signal that there is a backlog issue because latency rises long before jobs begin to fail or workers start to crash.

To set this up, click Queue time under Actions. Then select the following options and save the trigger:

• Namespace: Background
• Alert me when the: Mean
• Is above: more than, 30000ms
• Check Default email notifier

With this configuration, you will receive an email whenever the queue time on any monitored queue crosses its threshold. It will include all the information you need in order to identify the defective job.

Here’s an example of what this email looks like:

Wrap-Up

In this article, we started by explaining how AppSignal can help you monitor your Sidekiq job performance. Then, we demonstrated how you can set it up in a Ruby project and explored the three key signals that matter: job duration, queue latency, and error rate versus retry count. Finally, you learned how to set triggers and alerts for slow actions, error rate increases, and queue time thresholds.

This is just one of many ways AppSignal helps you build stable, scalable applications. If you want to create a more rounded observability setup for your background jobs, you can build on the knowledge gained here by exploring AppSignal's other monitoring features.