Why Your Sidekiq Jobs Are Slower Than You Think, and How AppSignal Fixes That

As developers, we sometimes find ourselves in situations where something feels slower than it should be. The first thing we tell ourselves is “Maybe I need a different tool.”

And sure, at times, it might be due to the tool, I’ll give you that. But most of the time, it comes down to how you’ve implemented certain features. That’s one of the downsides that comes with having too much freedom.

As for Sidekiq specifically, there are various reasons why your project might be getting slow or was slow to begin with. In this tutorial, we will cover a few of them:

• N+1 queries inside jobs
• Missing or misconfigured database indexes
• Synchronous external HTTP calls
• Too much work in a single job (no batching strategy)
• Inefficient Redis usage
• Retries and dead jobs consuming queue capacity
• Host usage

We will go through each of them, discuss why they occur, and demonstrate how to spot them easily using AppSignal. Our end goal is to spend less time going through code figuring out what's wrong and more time getting it fixed.

Jobs with N+1 Queries Inside Them

N+1 queries are a common issue in the Ruby space. In Sidekiq, they occur when you load a collection of records and each of them ends up having its own accompanying query.

This creates a very long and tedious loop. Imagine having 1,000 records and running a query on each; that’s bound to take a lot of time and resources, both of which are often limited.

Take this code, for example. Since customer.orders triggers a database query for every customer, the job generates an N+1 pattern:

class SendInvoicesJob
  include Sidekiq::Job
 
  def perform
    customers = Customer.active
 
    customers.each do |customer|
      InvoiceMailer.monthly_invoice(customer.orders).deliver_now
    end
  end
end

Instead of triggering a database query for every customer, you should load all the data you need upfront in a single query.

The AppSignal platform helps you spot this problem easily in the Issue list under the Performance tab. There’s also a warning box that notifies you whenever a query needs to be optimized.

These issues are displayed differently on the timeline, and identifying them prior to reading any code can be quite a timesaver.

Missing or Misconfigured Database Indexes

The main purpose of jobs is to handle certain tasks in the background while the app goes about its business. But if these jobs aren’t backed by the right indexes, each execution can force the database to scan large portions of a table just to find the records it needs.

For example, let’s look at a worker that processes pending orders:

class ProcessOrdersJob
  include Sidekiq::Job
 
  def perform
    Order.where(status: "pending")
         .where("created_at < ?", 1.day.ago)
         .find_each do |order|
      OrderProcessor.call(order)
    end
  end
end

If the status and created_at columns aren’t indexed, the database may end up performing expensive table scans on every run.

You can usually identify this issue in AppSignal by navigating to Performance > Slow Queries. When slow SQL statements appear in traces, it’s a good sign that the query is consuming most of the job runtime. AppSignal also ranks every query based on duration and frequency.

A query that takes 200 ms but only executes once a day won’t be that noticeable to your users, but a 100 ms query that runs on every request (for example, when someone looks at a product listing) can slow things down significantly at scale.

This is why frequency is also captured. A slow query that occurs often will show up first, and that’s usually the best place to start your investigation.

Synchronous External HTTP Calls

Background jobs sometimes need to communicate with external resources like payment gateways and email providers. Since you have no control over what’s going on on their side, the best you can do is monitor those external requests so you don’t start refactoring code when the problem isn’t actually on your end.

This kind of monitoring is also done by AppSignal, and you can find this data in the Slow events tab.

AppSignal traces external HTTP calls within jobs, showing you exactly how long each outbound request takes and which ones are causing bottlenecks.

Allocating Too Much Work in a Single Job (No Batching Strategy)

You might be tempted to pass large JSON payloads into a background job or create a worker that performs many unrelated tasks in the application. That may work fine for small personal projects, but for more serious ones, it’s something you may want to rethink.

Large payloads will increase the serialization time of your application and put extra pressure on your Redis memory usage. They will also slow down enqueue and dequeue operations. On top of that, having a single large job makes it difficult for Sidekiq to parallelize and retry properly because it is preoccupied with other things.

When job runtimes gradually increase, you can inspect the job metadata and transaction behavior to identify payload-related inefficiencies. You can also combine this with automated Redis monitoring, which we’ll cover in the next section.

As you can see in the image above, we are using five workers (that is, all available workers) for a single process. This means other jobs will have to wait until a worker becomes available.

Large spikes occupying all workers for long periods of time indicate that a job is handling too many responsibilities. You can easily identify this with the histogram.

Spikes in Redis Usage

Redis is essentially the backbone of most Sidekiq deployments, so its poor usage can slow down job processing even when you build code that is efficient.

Problems like this may show up when there are excessive round trips, store oversized values, repeat key lookups, or unnecessary Redis commands inside loops. In particular, too many Redis calls inside a loop can create a significant overhead.

AppSignal integrates with Redis instrumentation and records Redis activity as part of transaction traces. Any Redis queries performed by your Sidekiq job will appear in the Sidekiq dashboard on AppSignal.

Instead of assuming the database is to blame, you can quickly determine whether Redis interactions are consuming a significant portion of execution time. AppSignal also provides Redis monitoring for supported Redis clients and captures command durations that help expose bottlenecks. You can find this in the Slow queries performance panel.

Retries and Dead Jobs Consuming Queue Capacity

Misconfigured retry logic can flood your queues with failing jobs that end up competing with healthy ones. When a job fails and keeps retrying multiple times, it can start hoarding resources and delaying other jobs in the queue.

In Sidekiq, jobs fail silently, so it’s hard to identify them just by looking at the logs.

AppSignal's Sidekiq dashboards give you full visibility into retry storms through Overall job status. As a result, you can identify and fix root causes rather than just repeatedly clearing queues.

Besides retries, you can also see a number of dead jobs that can be investigated further. As you can spot in the picture, the dashboard provides more data, which can be used for comparisons and deeper analyses.

Could Your Host Be the Problem?

Host usage spikes happen when a server suddenly consumes significantly more CPU, memory, disk I/O, or network bandwidth than usual.

These spikes often don't last too long, so it’s difficult to identify them using logs alone. It can also be tricky to figure out whether the spike was caused by your job or by something at the infrastructure level.

AppSignal can help by collecting host metrics alongside application performance data. Instead of seeing a CPU graph in isolation, you can correlate resource usage with background job execution and other metrics.

Since AppSignal automatically monitors your host machine, no extra configuration is required. You can find host metrics in the Host Monitoring tab.

By clicking on a specific host, you can see when spikes have occurred across different resources and compare them with the periods where your jobs were running slowly.

Conclusion

In this tutorial, you’ve learned the reasons why your Sidekiq may not be such a good “sidekick” to your application, after all. You’ve seen how the speed of your job can be affected by either resource usage or programming anti-patterns.

To build on what you’ve learned in this tutorial, you can go ahead and set up anomaly detection. This will allow you to set thresholds on metrics that can help you identify issues before they become a major problem. You can also look into monitoring custom metrics for Sidekiq jobs in AppSignal.

If you don’t have an AppSignal account yet, you can start a free 30-day trial (no credit card required).