Solid Queue for Ruby on Rails: An Introduction (Rails 8.1)

Solid Queue is Rails’ default Active Job backend since Rails 8. It runs background jobs in your existing database — no Redis required — polling with FOR UPDATE SKIP LOCKED on MySQL and PostgreSQL (SQLite works without it). Choose it for fewer moving parts; choose Sidekiq for Redis-level throughput at very high scale.

In this first part of our two-part series, we’ll dig into Solid Queue’s internals: why 37signals built it, how jobs and workers coordinate through database tables, what makes its polling fast, and how the supervisor keeps jobs from getting lost.

Why Solid Queue for Ruby on Rails?

Since Rails 7, the team at 37signals has been on a quest to reduce the operational overhead of launching a new Rails application. As part of that, they made SQLite the default database for new Rails apps — even in production — and set out to eliminate the extra infrastructure dependencies that would undercut that default.

37signals had used Resque until then, and Resque requires Redis to function. So does Sidekiq, for that matter. To get rid of Redis, they needed a queuing system that relies only on your database — and that queuing system turned out to be Solid Queue.

That’s its main selling point: no additional dependencies — your database is the queue. Since Rails 8.0, Solid Queue has been the default Active Job backend, configured out of the box for production in every new application, and Rails 8.1 keeps that default. New Active Job features land on it first, too: Rails 8.1’s job continuations (ActiveJob::Continuable), which let long-running jobs checkpoint their progress and resume after an interruption, run through Solid Queue like any other job.

Being the Rails default sets a high bar. Solid Queue must provide the features Rails developers are used to from other background job systems, support every database that Rails works with, satisfy standard safety requirements — as in, it must never, ever lose jobs — and be fast enough to serve large production systems.

That’s a tall order! So, how does Solid Queue address all those requirements?

Solid Queue From The Top

The high-level architecture comes down to two components: jobs and workers.

Job is an ActiveRecord model, and every job you enqueue becomes a row that it manages. That’s not necessarily true for other Active Job backends — it’s how Solid Queue implements background jobs. Your own job classes stay plain Active Job classes, enqueued with methods like perform_later:

# app/jobs/my_job.rb
class MyJob < ApplicationJob
  queue_as :default
 
  def perform
    # Do something later
  end
end

Workers, as the name suggests, perform the actual work. You generally don’t create these directly; they are spawned based on how you configure your application. For example, to have your application spawn two workers listening to all and two specific queues respectively, you’d use the following configuration file:

# config/queue.yml
production:
  workers:
    - queues: "*"
    - queues: [default, critical]

Workers run as processes in the background, waiting for jobs to be assigned to them. Your database is the missing link between jobs and workers. Whenever Solid Queue does anything, one database table or other is involved. Solid Queue does a lot of things, so a lot of tables are needed.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
ActiveRecord::Schema[7.1].define(version: 1) do
  create_table "solid_queue_jobs", force: :cascade do |t|
    # ...
  end
 
  create_table "solid_queue_ready_executions", force: :cascade do |t|
    # ...
  end
 
  create_table "solid_queue_scheduled_executions", force: :cascade do |t|
    # ...
  end
 
  create_table "solid_queue_claimed_executions", force: :cascade do |t|
    # ...
  end
 
  create_table "solid_queue_blocked_executions", force: :cascade do |t|
    # ...
  end
 
  create_table "solid_queue_failed_executions", force: :cascade do |t|
    #...
  end
 
  # Lots more tables below...
end

The [7.1] in that header is the schema-format version solid_queue 1.7.0 still ships in its install template — not a sign of staleness.

The Life and Death of a SOLID Job

To understand what all those tables do and how they relate to the various features of Solid Queue, we can follow the life cycle of a job. When a user enqueues a job to be executed later — say, MyJob — a record is created in the solid_queue_jobs table. The record contains all the data required to execute the job — arguments, its name, the queue it is put in, and so forth. If the job is enqueued to run as soon as possible (rather than scheduled to run at some later point in time), an additional record is written to solid_queue_ready_executions.

For example, running MyJob.perform_later results in SQL like this (trimmed for readability):

INSERT INTO "solid_queue_jobs" ("queue_name", "class_name", "arguments", "priority", "active_job_id", "scheduled_at", "finished_at", "concurrency_key", "created_at", "updated_at")
  VALUES ('default', 'MyJob', '{"job_class": "MyJob","...",}', 0, '...', '2024-12-01 14:00:00', NULL, NULL, '2024-12-01 14:00:00', '2024-12-01 14:00:00')
  RETURNING "id"
INSERT INTO "solid_queue_ready_executions" ("job_id", "queue_name", "priority", "created_at")
  VALUES (1, 'default', 0, '2024-12-01 14:00:00')
  RETURNING "id"

Your workers poll this table for new records. A worker process that finds a new record will first claim it by writing a record to the solid_queue_claimed_executions table — we’ll learn why that is necessary later. Only then will the worker execute the job. Here is some heavily edited code to illustrate what is happening (much more is happening in the actual code). If you are curious about the nitty-gritty details, I highly recommend you check out the original source code.

class Worker
  def run
    loop do
      break if shutting_down?
 
      unless poll > 0
        # Polling interval is configurable and defaults to 0.1 seconds
        sleep(polling_interval)
      end
    end
  end
 
  def poll
    # Claim jobs and then execute claimed jobs.
    claim_executions.then do |executions|
      executions.each do |execution|
        # Actually execute the job
      end
    end
  end
 
  def claim_executions
    # Query the ready executions table and claim a job for execution.
    with_polling_volume do
      SolidQueue::ReadyExecution.claim
    end
  end
end

A note on that polling interval: workers default to polling every 0.1 seconds (dispatchers poll every second, and the scheduler every 5 seconds), while the config/queue.yml that Rails generates sets workers to a more relaxed 1 second.

Once a worker finishes a job, it removes the corresponding record from solid_queue_claimed_executions and stamps the job’s finished_at timestamp in solid_queue_jobs — the solid_queue_ready_executions record was already removed when the job was claimed. Finished jobs stick around for inspection until a recurring cleanup task clears them, one day later by default. That’s all there is to it — polling some tables, creating and removing records. Not so tricky, right? It would be, if there weren’t critical non-functional requirements to consider, too.

On Performance

To achieve production-ready performance, Solid Queue uses ingenious database design. You may have wondered why workers poll solid_queue_ready_executions rather than solid_queue_jobs. The additional table seems redundant at first glance.

Consider that solid_queue_jobs may contain thousands or millions of records, and querying that pile of data takes time. In comparison, solid_queue_ready_executions is tiny, as it only contains records for jobs that must be executed right now! That leads to some serious speedup.

The introduction of additional tables also simplifies queries. Workers only use two different queries for polling. They either poll all queues or specific ones. That, in turn, allows for some nice covering indices.

SELECT job_id
  FROM solid_queue_ready_executions
  WHERE queue_name = "default"
  ORDER BY priority ASC, job_id ASC
  LIMIT 4
  FOR UPDATE SKIP LOCKED

# Indices for polling solid queue ready executions
create_table "solid_queue_ready_executions", force: :cascade do |t|
  t.index [ "priority", "job_id" ], name: "index_solid_queue_poll_all"
  t.index [ "queue_name", "priority", "job_id" ], name: "index_solid_queue_poll_by_queue"
end

All that still wouldn’t be enough to achieve outstanding performance. Traditionally, queuing systems that rely on polling tables have had a significant problem. One worker would block all others while querying and updating the polling table.

Consider the following query, typical of such a system:

SELECT id
  FROM jobs
  WHERE queue = "default"
  AND claimed = 0
  ORDER BY priority, id
  LIMIT 2
  FOR UPDATE;

The FOR UPDATE statement locks the rows selected by the query. This is necessary to avoid nasty race conditions, such as multiple workers grabbing the same job. But that also means that any worker running this query would block read access to the table. Thus, other workers would have to wait for that query to finish. The polling table becomes a bottleneck that hinders rapid job execution.

Luckily, modern databases (PostgreSQL >= 9.5, MySQL >= 8.0) solve this problem. The SKIP LOCKED statement allows the database to lock only the records that are being updated. The rest of the table remains unlocked and free to be polled concurrently.

SQLite does not support SKIP LOCKED, so worker processes must queue up. In most cases, this shouldn’t be an issue — SQLite writes are fast, as the database lives on local disk. Even so, this is a limitation that you should be aware of.

Safety First

We’ve spent some time discussing solid_queue_ready_executions, but another table is instrumental for ensuring that Solid Queue functions reliably. A key requirement of any queuing system is that any job being enqueued is executed at least once. In other words, jobs must never be lost.

Without additional safety measures, this could quickly happen. Imagine that a worker starts working on a job and, in doing so, claims that job. This is necessary to avoid multiple workers running a job simultaneously.

Imagine that suddenly, this worker process dies without finishing execution. Your machine might crash, and the OS may kill the worker for consuming too much memory — accidents happen, you know. The job it claimed will remain stuck forever because no other workers can grab it. Thus, it will never be executed, and your users will be sad and angry. The end.

That is, unless we add additional safety measures. Solid Queue solves this problem by introducing yet more tables — solid_queue_claimed_executions and solid_queue_processes.

ActiveRecord::Schema[7.1].define(version: 1) do
  create_table "solid_queue_claimed_executions", force: :cascade do |t|
    t.bigint "job_id", null: false
    t.bigint "process_id"
    # ...
  end
 
  create_table "solid_queue_processes", force: :cascade do |t|
    t.datetime "last_heartbeat_at", null: false
    t.integer "pid", null: false
    #  ...
  end
  # ...
end

We’ve already mentioned solid_queue_claimed_executions. Here is what claiming a job means: the job’s record in solid_queue_ready_executions is deleted, and a new record is inserted into solid_queue_claimed_executions in the same transaction. This record contains the job_id of the job being claimed and the id of the worker process that makes the claim. There is no flag on the job itself — the claim is that move from one table to the other.

So, what is the solid_queue_processes table good for? Any worker process will create and periodically update a record in this table by setting last_heartbeat_at. That alone wouldn’t solve our problem, though.

We need another process to keep track of running processes: the so-called supervisor. This process runs in the background and periodically checks solid_queue_processes. A record with a last_heartbeat_at older than a threshold — which defaults to 5 minutes — indicates that the corresponding worker has met a tragic fate.

If such a record is found, the supervisor jumps into action. First, it removes the record from solid_queue_processes. Then, it marks any jobs previously claimed by the now-deceased worker as up-for-grabs. Thus, other workers can claim them, avoiding the stuck-job situation.

Solid Queue vs Sidekiq: Which Should You Use?

The most common question about Solid Queue is whether it replaces Sidekiq. A short decision rule:

• Choose Solid Queue when you want fewer moving parts and your job volume fits comfortably within your database’s write capacity — which covers most Rails applications.
• Choose Sidekiq when you need Redis-level throughput — millions of jobs per hour — or you depend on its ecosystem of extensions.
• Already running Redis anyway? The infrastructure argument for Solid Queue weakens, but a default maintained by the Rails team remains the lower-maintenance choice for a new application.

Many Rails 8 apps start on Solid Queue and never need to switch — and if you do outgrow it, moving to Sidekiq is an adapter change rather than a rewrite, because Active Job shields your job classes from the backend. For a feature-by-feature comparison of the two systems, see Sidekiq vs Solid Queue in Rails 8 in our Sidekiq introduction.

Monitoring Solid Queue in Production

A database-backed queue fails in database-shaped ways, and three failure modes deserve attention before you ship:

• Silent failures: a job that raises lands in solid_queue_failed_executions and waits there for manual inspection — nothing retries or pages you by default.
• Queue latency vs. job runtime: jobs can execute in milliseconds yet spend minutes queued behind a backlog, so measuring only runtime hides the delay your users experience.
• Database contention: workers, dispatchers, and your application all share one database, so a ballooning queue table competes with user-facing queries for writes.

Because Solid Queue lives in your database, a stuck worker or ballooning queue surfaces as database load — not just late jobs — so you need queue-level visibility. AppSignal instruments Solid Queue out of the box, tracking job runtimes, queue latency, and failures alongside the rest of your Rails app.

AppSignal for Ruby picks up Solid Queue automatically once installed — the Solid Queue integration docs cover setup — and part two’s monitoring section shows the resulting dashboards and alerts in action.

More to Discover in Solid Queue

That covers the core machinery — but there is more to discover. Part two of this series digs into scheduling, recurring cron-style tasks, and concurrency controls. Solid Queue also ships job batches — jobs grouped and tracked as one unit, backed by a batch_id column on every job and batch maintenance built into the dispatcher — which part two doesn’t yet cover. Plenty of reasons to keep exploring.