Elom Gomez | August 15, 2022

We’ve have all run into situations where we’d like to disable or cancel an errant Sidekiq job in production. The current solutions involve dumping the deploy queue for the specific job or deploying a hot fix to production to disable the job. Initially, these solutions tend to solve your problem, but as your app grows and your job queue gets complicated, you’ll want a solution that is more robust and almost immediate.

In this post, we will learn how we built a Sidekiq Client Middleware to kill Sidekiq jobs in production without relying on deploys.

Sidekiq Client Middleware

We currently make use of the amazing flipper gem at PlanetScale to feature flag new features being built. One of the major pros of Flipper is that it allows us to enable/disable different code paths in production without deploys. Based on this, we decided to use its feature flagging capabilities to help us disable our jobs in production. We created a middleware, lib/sidekiq_middleware/sidekiq_jobs_flipper.rb, which you can find below:

module SidekiqMiddleware
  class SidekiqJobsFlipper
    def call(worker_class, job, queue, redis_pool)
      # return false/nil to stop the job from going to redis
      klass = worker_class.to_s
      if Flipper.enabled?("disable_#{klass.underscore.to_sym}")
        return false
      end

      yield
    end
  end
end

The Sidekiq client middleware runs when pushing a job to Redis. The following snippet from the middleware allows us to short circuit any jobs that have been feature flagged to be disabled:

if Flipper.enabled?("disable_#{klass.underscore.to_sym}")
  return false
end

Example usage

If you want to disable jobs in production in your own application, you can use this middleware.

1. Create a new middleware file and paste in the middleware code from the previous section.

2. Add the middleware to your Sidekiq configurations in config/initializers/sidekiq.rb:

 Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add(SidekiqMiddleware::SidekiqJobsFlipper)
  end
 end

3. To disable a job in production, you would .underscore the job class name and prefix it with disable_. Let’s take the InvoiceJob below as an example:

class InvoiceJob
  include Sidekiq::Worker

  def perform(...)
    ...
  end
end

In this case, the class name InvoiceJob becomes disable_invoice_job.

4. To disable InvoiceJob, run the following command in the console:

Flipper.enable("disable_invoice_job")