Sidekiq Unique Jobs: Three Ways to Avoid Duplicates

To avoid duplicate Sidekiq jobs, you have three options: guard enqueues yourself with database flags or a queue scan, use Sidekiq Enterprise’s unique_for option, or add the free sidekiq-unique-jobs gem, which locks jobs in Redis by class and arguments. On Sidekiq 8, sidekiq-unique-jobs 8 with lock: :until_executed covers most cases.

Chances are, if you write Ruby, you use Sidekiq to handle background processing. (New to it? Start with our introduction to Sidekiq for Ruby on Rails.) Folks use background jobs for all kinds of work: some crunch numbers, some dispatch welcome emails, and some schedule data syncing. Whatever your case may be, you will eventually run into a requirement to avoid duplicate jobs — two jobs doing the exact same thing. If you come from Active Job or another background job framework, most of the tips covered here apply there as well.

Every snippet in this post was re-verified against a live Redis on Sidekiq 8.1.7, sidekiq-unique-jobs 8.1.0, and Ruby 3.4.

Why De-Duplicate Jobs?

Imagine a scenario where your job looks like the following:

class BookSalesWorker
  include Sidekiq::Job
 
  def perform(book_id)
    crunch_some_numbers(book_id)
 
    upload_to_s3
  end
 
  ...
end

The BookSalesWorker always does the same thing — it queries the DB for a book based on the book_id, fetches the latest sales data to calculate some numbers, and uploads them to a storage service. Every time a book is sold on your website, this job gets enqueued.

One aside: the worker does include Sidekiq::Job, where older codebases write include Sidekiq::Worker. Both work on Sidekiq 8 — Sidekiq::Worker and Sidekiq::Job are the same module under two names (Sidekiq::Worker.equal?(Sidekiq::Job) returns true), with Job as the preferred spelling.

Now, what if you got 100 sales at once? You’d have 100 of these jobs doing the exact same thing. Maybe you are fine with that. You don’t care about S3 writes that much, and your queues aren’t congested, so you can handle the load. But, “does it scale?”™️

Well, definitely not. If you start receiving more sales for more books, your queue quickly piles up with unnecessary work. With 100 identical jobs per book and 10 books selling in parallel, you are 1,000 jobs deep in your queue, where 10 jobs — one per book — would do the same work.

Which Approach Should You Choose?

Here is the whole decision in one table:

If your app runs on Solid Queue rather than Sidekiq, the framework has a built-in answer to the same problem — see concurrency controls in our Solid Queue deep dive.

1. DIY Way

If you are not a fan of external dependencies and complex logic, you can add a custom solution to your codebase. I created a sample repo to try out our examples first-hand. There will be a link in each approach to the example.

1.1 One Flag Approach

You can add one flag that decides whether to enqueue a job or not. One might add a sales_enqueued_at to their Book table and maintain that one. For example:

module BookSalesService
  module_function
 
  def schedule_with_one_flag(book)
    # Check if the job was enqueued more than 10 minutes ago
    if book.sales_enqueued_at < 10.minutes.ago
      book.update(sales_enqueued_at: Time.current)
 
      BookSalesWorker.perform_async(book.id)
    end
  end
end

That means no new jobs will be enqueued until 10 minutes have passed since the last job was enqueued. After that, we update the sales_enqueued_at and enqueue a new job.

Another thing you can do is set one flag that is a boolean, e.g. crunching_sales. You set crunching_sales to true before the first job is enqueued. Then, you set it to false once the job is complete. All other jobs that try to get scheduled will be rejected until crunching_sales is false.

You can try this approach in the example repo I created.

1.2 Two Flags Approach

If “locking” a job from being enqueued for 10 minutes sounds too scary, but you are still fine with extra flags in your code, then the next suggestion might interest you.

You can add another flag to the existing sales_enqueued_at — the sales_calculated_at. Then our code will look something like this:

module BookSalesService
  module_function
 
  def schedule_with_two_flags(book)
    # Check if sales are being calculated right now
    if book.sales_enqueued_at <= book.sales_calculated_at
      book.update(sales_enqueued_at: Time.current)
 
      BookSalesWorker.perform_async(book.id)
    end
  end
end
 
class BookSalesWorker
  include Sidekiq::Job
 
  def perform(book_id)
    book = Book.find(book_id)
 
    crunch_some_numbers(book_id)
 
    upload_to_s3
 
    # New addition
    book.update(sales_calculated_at: Time.current)
  end
 
  ...
end

To try it out, check out the instructions in the example repo.

Now we control a portion of the time between when a job is enqueued and when it finishes. In that portion of time, no job can be enqueued. While the job is running, the sales_enqueued_at will be larger than sales_calculated_at. When the job finishes running, the sales_calculated_at will be larger (more recent) than the sales_enqueued_at, and a new job will get enqueued.

Using two flags might be interesting because you can show the last time those sales numbers got updated in the UI. Then the users who read them can see how recent the data is. A win-win situation.

When Flags Are Enough

It might be tempting to create solutions like these in times of need, but to me, they look a bit clumsy, and they add some overhead. I would recommend this route for a simple use case, but as soon as it proves complex or insufficient, I’d urge you to try out the other options.

A huge con of the flag approach is that you lose all the jobs that tried to enqueue during those 10 minutes. A huge pro is that you are not bringing in dependencies, and it alleviates the job count in your queues pretty quickly.

1.3 Traversing The Queue

Another approach you can take is to create a custom locking mechanism to prevent the same jobs from enqueueing. We will check the Sidekiq queue we are interested in and see whether the job (worker) is already there. The code will look something like this:

module BookSalesService
  module_function
 
  def schedule_unique_across_queue(book)
    queue = Sidekiq::Queue.new('default')
 
    queue.each do |job|
      return if job.klass == BookSalesWorker.to_s &&
        job.args == [book.id]
    end
 
    BookSalesWorker.perform_async(book.id)
  end
end

In this example, we are checking whether the 'default' queue has a job with the class name of BookSalesWorker. We are also checking whether the job arguments match the book ID. If the BookSalesWorker job with the same book ID is in the queue, we return early and don’t schedule another one. The Sidekiq::Queue API works the same way on Sidekiq 8 as it did when this post was first written.

Some duplicates can still get scheduled if you enqueue jobs too fast, because the queue starts out empty. The exact thing happened to me when testing it locally with:

100.times { BookSalesService.schedule_unique_across_queue(book) }

You can try it out in the example repo.

The good thing about this approach is that you can traverse all queues to search for an existing job if you need to. The con is that you can still get duplicate jobs when your queue is empty and you schedule a lot of them at once. Also, you are potentially traversing all the jobs in the queue before scheduling one, and that can get costly depending on the size of your queue.

2. Upgrading to Sidekiq Enterprise

If you or your organization has some money lying around, you can upgrade to the Enterprise version of Sidekiq. It starts at $269 per month for 100 worker threads (sidekiq.org, checked August 2026), and it comes with a built-in unique jobs feature. I don’t have Sidekiq Enterprise, so unlike everything else in this post, the claims in this section come from the official documentation rather than a verified test run.

You first enable the unique jobs subsystem in an initializer:

# config/initializers/sidekiq.rb
Sidekiq::Enterprise.unique! unless Rails.env.test?

Then each worker declares how long it should stay unique:

class BookSalesWorker
  include Sidekiq::Job
  sidekiq_options unique_for: 10.minutes
 
  def perform(book_id)
    crunch_some_numbers(book_id)
 
    upload_to_s3
  end
 
  ...
end

And that’s it. You get a similar result to the ‘One Flag Approach’ section: the job stays unique for 10 minutes, meaning no other job with the same class, arguments, and queue can be pushed to Redis in that time period. The unique_until: option controls when the lock releases — on success by default (unique_until: :success), or as soon as the job starts with unique_until: :start.

A pretty cool one-liner, huh? If you have Sidekiq Enterprise and you learned about this feature here, I am truly glad I helped. Most of us are not going to use it, though, so on to the next solution.

3. sidekiq-unique-jobs To The Rescue

The sidekiq-unique-jobs gem brings unique jobs to open-source Sidekiq, with a lot of locking and conflict-resolution options — probably more than you need. Its locks run as Lua scripts inside Redis, where they execute atomically. Version 8 of the gem supports both Sidekiq 7 and Sidekiq 8 (its gemspec allows sidekiq >= 7.0.0, < 9.0.0).

To get started, put the sidekiq-unique-jobs gem into your Gemfile and run bundle. Then comes the step that decides whether the gem does anything at all.

Configuring the sidekiq-unique-jobs Middleware

The gem does its locking through Sidekiq middleware, and that middleware is not installed for you. Without it, lock: and on_conflict: options on your workers are inert — Sidekiq accepts them silently and enqueues every duplicate. In my test against Sidekiq 8.1.7, five identical pushes all enqueued five jobs until the middleware was configured; with it, one job enqueued and four were rejected. If you remember one thing from this post, make it this initializer:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
 
  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end
 
  SidekiqUniqueJobs::Server.configure(config)
end
 
Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

The client middleware takes the lock when a job is pushed (that is what stops duplicates at enqueue time — your web process needs it too, hence the configure_client block), the server middleware releases locks around job execution, and SidekiqUniqueJobs::Server.configure wires up the gem’s lock maintenance inside the Sidekiq process.

With that in place, configure your worker as shown:

class UniqueBookSalesWorker
  include Sidekiq::Job
 
  sidekiq_options lock: :until_executed,
                  on_conflict: :reject
 
  def perform(book_id)
    book = Book.find(book_id)
 
    logger.info "I am a Sidekiq Book Sales worker - I started"
    sleep 2
    logger.info "I am a Sidekiq Book Sales worker - I finished"
 
    book.update(sales_calculated_at: Time.current)
    book.update(crunching_sales: false)
  end
end

The gem has a lot of options, but I kept it to this pair:

sidekiq_options lock: :until_executed, on_conflict: :reject

The lock: :until_executed option locks the first UniqueBookSalesWorker job until it is executed. The lock key — the lock_digest you will see in the job payload — is computed from the worker class and its arguments, so perform_async(1) and perform_async(2) hold separate locks. With on_conflict: :reject, every job that conflicts with a held lock is pushed to the dead queue. What we achieve here is similar to our DIY examples from earlier sections.

A slight improvement over those DIY examples is that we get a log of what happened. To get a sense of how it looks, run the following:

5.times { UniqueBookSalesWorker.perform_async(Book.last.id) }

Only one job fully executes; the other four are sent off to the dead queue, where you can retry them. This differs from our DIY examples, where duplicate jobs were dropped without a trace. Each rejection is logged by the gem — in my run, four lines of Adding dead UniqueBookSalesWorker job 8949b7ed6cb337cf1284d244 (each with its own job ID). If you would rather drop duplicates than dead-set them, on_conflict: :log skips the push and prints Skipping job with id (5c2d62a9ba7d605a35c06b56) because lock_digest: (uniquejobs:d62b2adb2dff3209a7459246b09765be) already exists instead. Those two log lines are also your first debugging stop when locks misbehave.

There are many more options for locking and conflict resolution — consult the gem’s documentation for your specific use case.

With on_conflict: :reject, duplicate jobs land in the dead set silently — and a misconfigured lock sends legitimate work there too. AppSignal’s Sidekiq integration graphs queue lengths, dead jobs, and per-worker failures automatically, so you notice when deduplication starts eating jobs you needed.

Locks and Changelogs in the Sidekiq Web UI

A great thing about this gem is that you can view the locks and the history of what went down in your queues. Add the following lines to your config/routes.rb:

# config/routes.rb
require 'sidekiq_unique_jobs/web'
 
Rails.application.routes.draw do
  mount Sidekiq::Web, at: '/sidekiq'
end

This includes the original Sidekiq web UI, plus two more pages — one for job locks and the other for the changelog. This is how it looks:

Notice the two new pages, “Locks” and “Changelogs”. A pretty cool feature.

You can try all of this in the example project, where the gem is installed and ready to go.

Alternative: activejob-uniqueness

If you rely heavily on ActiveJob, the activejob-uniqueness gem is worth a look: the idea is similar, but instead of custom Lua scripts, it uses Redlock to lock items in Redis, and its less verbose unique :until_executed declaration (verified on version 0.4.0) sits right in your job class:

class BookSalesJob < ActiveJob::Base
  unique :until_executed
 
  def perform
    ...
  end
end

Final Thoughts

I hope you gained some knowledge on how to deal with duplicate jobs in your app. To recap the three ways: database flags are free but lossy, Sidekiq Enterprise’s unique_for is a one-liner if you already pay for it, and the sidekiq-unique-jobs gem covers most cases — as long as its middleware is configured. I definitely had fun researching and playing around with the different solutions, and if you didn’t find exactly what you were looking for, I hope some of the examples inspired you to create something of your own.

Here’s the example project with all the code snippets.

I will see you in the next one, cheers.

P.S. If you’d like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!