Work with two separate redis instances with sidekiq?

Question

Good afternoon,

I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes). Howev

Answer 1

As carols10cents says its pretty simple but as I always like to encapsulate the capability and be able to reuse it in other projects I updated an idea from a blog from Hotel Tonight. This following solution improves upon Hotel Tonight's that does not survive Rails 4.1 & Spring preloader.

Currently I make do with adding the following files to lib/remote_sidekiq/:

remote_sidekiq.rb

class RemoteSidekiq
  class_attribute :redis_pool
end

remote_sidekiq_worker.rb

require 'sidekiq'
require 'sidekiq/client'

module RemoteSidekiqWorker
  def client
    pool = RemoteSidekiq.redis_pool || Thread.current[:sidekiq_via_pool] || Sidekiq.redis_pool
    Sidekiq::Client.new(pool)
  end

  def push(worker_name, attrs = [], queue_name = "default")
    client.push('args' => attrs, 'class' => worker_name, 'queue' => queue_name)
  end
end

You need to create a initializer that sets redis_pool

config/initializers/remote_sidekiq.rb

url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

EDIT by Aleks:

In never versions of sidekiq, instead of lines:

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

use lines:

redis_remote_options = {
  namespace: "yournamespace",
  url: ENV.fetch("REDISCLOUD_URL")
}

RemoteSidekiq.redis_pool = Sidekiq::RedisConnection.create(redis_remote_options)

You can then simply the include RemoteSidekiqWorker module wherever you want. Job done!

**** FOR MORE LARGER ENVIRONMENTS ****

Adding in RemoteWorker Models adds extra benefits:

1. You can reuse the RemoteWorkers everywhere including the system that has access to the target sidekiq workers. This is transparent to the caller. To use the "RemoteWorkers" form within the target sidekiq system simply do not use an initializer as it will default to using the local Sidekiq client.
2. Using RemoteWorkers ensure correct arguments are always sent in (the code = documentation)
3. Scaling up by creating more complicated Sidekiq architectures is transparent to the caller.

Here is an example RemoteWorker

class RemoteTraceWorker
  include RemoteSidekiqWorker
  include ActiveModel::Model

  attr_accessor :message

  validates :message, presence: true

  def perform_async
    if valid?
      push(worker_name, worker_args)
    else
      raise ActiveModel::StrictValidationFailed, errors.full_messages
    end
  end

  private

  def worker_name
    :TraceWorker.to_s
  end

  def worker_args
    [message]
  end
end