Skip to content

Grokem · https://www.grokem.com/programming/web-development/backend/background-jobs-in-a-rails-backend-when-to-use-them-and-start-with-sidekiq

BackendHow-to guideIntermediate level

Background jobs in a Rails backend: when to use them and start with Sidekiq

Learn which work belongs in a background job and how to move it there with Active Job. You start with an in-process queue, then switch to Sidekiq and Redis with a separate worker process.

Pankaj Nigam

Technical Writer

Published
  • 30 min to complete
  • 0 sec

Tested with
  • Ruby on Rails
  • Active Job
  • Sidekiq
  • Redis
Your progress
  1. Background jobs in a Rails backend: when to use them and start with Sidekiq6 min(finished)
    0%

Before you start

4
  • A working Rails application you can run locally
  • Basic knowledge of Ruby classes and the Rails console
  • Bundler installed, to add gems and run commands with bundle exec
  • For the Sidekiq step: a Redis server you can reach (a local install or a hosted add-on)

A web request should return fast. When a request also has to send email, clean up records or call a slow service, you move that work out of the request and into a background job. The web process puts a description of the work on a queue. A separate worker picks it up and runs it.

Move work to a job when it matches one or more of these cases:

  • The user does not need the result to receive a response.
  • The work is slow enough to hold the request open.
  • The work should run later, for example in a week or tomorrow at noon.
  • The work may fail and you want control over what happens next, such as retrying or discarding it.

Keep work in the request when the response depends on its result.

In Rails you write the job once with Active Job, a wrapper that lets you change the queuing backend later. This guide uses two backends. First you use the in-process one that ships with Rails, so you can learn the job API with no extra services. Then you switch to Sidekiq, which stores jobs in Redis and runs them in a separate worker process.

The in-process queue keeps jobs in memory and runs inside the Rails process. That is fine for learning, but it does not support priority, timeout or retry. Sidekiq is faster than the alternatives described here but requires thread-safe code.

This guide does not cover every Sidekiq option. For anything not shown, such as the contents of config/sidekiq.yml, check the Sidekiq wiki linked in the references.

Step 1 of 11: Choose the work to move out of the request

Pick one action in your app that fits the cases in the overview, such as a welcome email or a cleanup task. Start with one so you can check the full path from request to worker.

The flow you build looks like this: the web process enqueues a job, the queue holds it, and a worker runs it later.

Rails web process(request returns at once)Redis queue(critical, default)Sidekiq worker(runs perform)perform_laterpicks up jobfast response
The web process enqueues the job and responds while a worker runs it.

Keep the job small. It should receive identifiers or simple values and do the slow part itself.

Step 2 of 11: Generate the job

Run the Rails generator with a queue name. The generator creates the job class in app/jobs, so you start from the structure Active Job expects.

bash
rails g job my_todo --queue critical

Step 3 of 11: Write the perform method

Open app/jobs/my_todo_job.rb and put the work inside perform. Active Job calls perform when a worker runs the job, so everything slow belongs there.

The sleep 5 stands in for slow work, so you can see that the request does not wait for it.

ruby
# app/jobs/my_todo_job.rb
class MyTodoJob < ApplicationJob
  queue_as :critical

  def perform(*args)
    Rails.logger.info "MyTodoJob start #{args}"
    sleep 5
    Rails.logger.info "MyTodoJob end"
  end

Step 4 of 11: Enqueue the job from your code

Call perform_later where the request handler would otherwise do the work. This puts the job on the queue and returns at once, so the response is not delayed.

Use set with wait or wait_until to delay the job. Use perform_now to run it immediately in the current process, which is handy in the console. Note that in tests, delayed jobs are performed immediately.

ruby
MyTodoJob.perform_later 'first', 'second'
MyTodoJob.set(wait: 1.week).perform_later 'first'
MyTodoJob.set(wait_until: Date.tomorrow.noon).perform_later 'first'
MyTodoJob.perform_now 'first'

Step 5 of 11: Pass safe arguments

With Active Job you can pass an entire ActiveRecord object, because GlobalID serializes and restores it for you. This keeps perform simple.

If you call a job class that does not inherit from Active Job, such as a plain Sidekiq worker, pass the object id instead. Otherwise you get an error like Unsupported argument type, for example when calling UserMailer.signup(move).deliver_later with an object instead of an id.

Step 6 of 11: Set the in-process queue adapter

Set the adapter to :async in config/application.rb. The default is :inline, and the :async adapter is the better choice for local work because it runs jobs inside the Rails process without blocking the caller.

Remember the limits: both :inline and :async do not support priority, timeout or retry. Because jobs live in memory and run with Rails, a debugger breakpoint inside a job stops the Rails process.

ruby
# config/application.rb, inside your Application class
config.active_job.queue_adapter = :async

Step 7 of 11: Switch to Sidekiq

Add the gem to the Gemfile, then install it. Sidekiq moves jobs out of the Rails process into Redis, so a separate worker can run them.

Then point Active Job at Sidekiq. The optional queue_name_prefix changes queue names, so default becomes mysite_default and mailer becomes mysite_mailer.

ruby
# Gemfile
gem 'sidekiq'
bash
bundle install
ruby
# config/application.rb, inside your Application class
config.active_job.queue_adapter = :sidekiq
# optional: prefixes every queue name
config.active_job.queue_name_prefix = 'mysite'

Step 8 of 11: Provide Redis and limit its connections

Make a Redis server available, for example the Heroku Redis add-on or your own install. Sidekiq stores the queue in Redis, so without it jobs cannot be enqueued.

Watch the connection limit. Sidekiq usually needs concurrency + 5 connections to Redis. If your plan allows only 20 connections, lower the Sidekiq concurrency accordingly. How to point Sidekiq at your Redis server is not shown here; follow the Sidekiq wiki.

Step 9 of 11: Start the worker process

Run Sidekiq as its own process next to the web server. The worker reads jobs from the named queues, so it must be told which queues to watch.

On a platform that uses a Procfile, declare the worker next to the web process. You can also put the queues in a config file passed with -C, as in the second Procfile line.

bash
bundle exec sidekiq -q default -q mailers
bash
# Procfile
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml

Step 10 of 11: Control retries and scheduling with a native worker

If you use a plain Sidekiq worker, include Sidekiq::Worker and set sidekiq_options retry: 0 to turn off automatic retries. Use this when a failed run should not repeat.

With Active Job, discard_on discards a job without retries; see the Rails API page in the references for its arguments. The scheduling calls below mirror the Active Job ones.

ruby
class CleanupWorker
  include Sidekiq::Worker
  sidekiq_options retry: 0

  def perform(name)
    Rails.logger.info "Cleanup for #{name}"
  end

CleanupWorker.perform_async 'duke'
CleanupWorker.perform_in 3.hours, 'duke'
CleanupWorker.perform_at 3.hours.from_now, 'duke'
# run in the current process, without the queue
CleanupWorker.new.perform 'duke'

Step 11 of 11: Enqueue a job and watch the worker

Start the Rails console, enqueue the job and watch the worker output. This confirms the whole path from web process to queue to worker.

bash
bundle exec rails console
ruby
MyTodoJob.perform_later 'hello'

Expected result

The `perform_later` call returns immediately with the enqueued job object, without waiting five seconds. A few seconds later the log shows "MyTodoJob start" followed by your arguments, then "MyTodoJob end". With the Sidekiq adapter, the worker process picks the job up, not the Rails console process. With `perform_now`, the call blocks until the job ends.

Frequently asked questions

Do I need Sidekiq to start with background jobs?

No. Active Job with the :async adapter runs jobs inside the Rails process, which is enough to learn the API. It does not support priority, timeout or retry, so move to Sidekiq when you need them.

What is the difference between perform_later and perform_now?

perform_later puts the job on the queue and returns. perform_now runs it immediately in the current process.

How do I delay a job?

Use set with wait or wait_until, for example MyTodoJob.set(wait: 1.week).perform_later. For plain Sidekiq workers use perform_in or perform_at.

Can I pass a model object to a job?

With Active Job yes, because GlobalID restores it. For classes that do not inherit from Active Job, pass the id.

Why does Sidekiq need Redis?

Sidekiq keeps its queues in Redis, so the web process and the worker process can share them.

How do I stop a job from retrying?

For a plain Sidekiq worker, set sidekiq_options retry: 0. With Active Job, use discard_on; see the Rails API page for its arguments.

References

From Grokem: https://www.grokem.com/programming/web-development/backend/background-jobs-in-a-rails-backend-when-to-use-them-and-start-with-sidekiq

Finished this article?

How did this strike you?

Be the first to react – it is anonymous.

Community

Questions about this?

Ask in Questions and help and get help from other readers.

Ask a questionBrowse Questions and help

Comments

No comments yet. Share what worked for you, or ask a question.

Keep learning

Picked for this article

Related reading, discussions and learning paths on the same subject.

All in Backend

From the community

Learning paths

Learning pathIntermediate

Building and calling APIs

Design an HTTP API people can rely on, read its status codes, choose between REST, GraphQL and gRPC, receive webhooks safely and call a paginated API with retries.

5 lessonsabout 50 minStart

Learning pathBeginner

Infrastructure basics for data work

The ground a data project stands on: the command line, how a name finds a server, packaging an app in a container, where files and databases live, keeping secrets out of code, backups you have tested and a cloud bill you control.

7 lessonsabout 15 minStart

Choose what we may use. You can change this at any time with “Privacy choices” at the foot of every page. Details are in the Cookie policy and the Privacy policy.

Strictly necessary

Remembers your theme, the lessons you marked as done and this choice. It stays on your device, tracks nothing and cannot be switched off.

Always on
What exactly is collected, and why

Google Analytics

Collects
The pages you view, your device and browser type, your language, how you reached the site and your approximate location (country and city, worked out from your IP address).
Why
To understand which pages and lessons are used, so we can improve them.
Stored
Two cookies, _ga and _ga_<id>, for up to 2 years. Google acts for us; the data is processed by Google, including outside your country.

Grokem’s own counts

Collects
The pages you open and how long you stay, how far you scroll, where you came from, your device type, browser, screen size, language and time zone. Your IP address is turned into a code that changes every day; it is never stored as it is.
Why
To see how long lessons are read and where readers leave. Not shared with anyone.
Stored
A random visitor ID, a visit ID and the time of your last page view in your browser’s storage (gr_vid, gr_sid, gr_last). No name or email is attached.