Work with two separate redis instances with sidekiq?

Question

Good afternoon,

I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes) . However, I'd like to occasionally be able to push jobs onto app2 's queue from app1 .

From a simple queue/push perspective, it would be easy to do this if app1 did not have an existing Sidekiq/Redis stack:

# In a process, far far away

# Configure client 
Sidekiq.configure_client do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

# Push jobs without class definition 
Sidekiq::Client.push('class' => 'Example::Workers::Trace', 'args' => ['hello!'])

# Push jobs overriding default's 
Sidekiq::Client.push('queue' => 'example', 'retry' => 3, 'class' =>     'Example::Workers::Trace', 'args' => ['hello!'])

However given that I would already have called a Sidekiq.configure_client and Sidekiq.configure_server from app1 , there's probably a step in between here where something needs to happen.

Obviously I could just take the serialization and normalization code straight from inside Sidekiq and manually push onto app2 's redis queue, but that seems like a brittle solution. I'd like to be able to use the Client.push functionality.

I suppose my ideal solution would be someting like:

SidekiqTWO.configure_client { remote connection..... } SidekiqTWO::Client.push(job....)

Or even:

$redis_remote = remote_connection.....

Sidekiq::Client.push(job, $redis_remote)

Obviously a bit facetious, but that's my ideal use case.

Thanks!

Answer 1

So one thing is that According to the FAQ , "The Sidekiq message format is quite simple and stable : it's just a Hash in JSON format." Emphasis mine-- I don't think sending JSON to sidekiq is too brittle to do. Especially when you want fine-grained control around which Redis instance you send the jobs to, as in the OP's situation, I'd probably just write a little wrapper that would let me indicate a Redis instance along with the job being enqueued.

For Kevin Bedell's more general situation to round-robin jobs into Redis instances, I'd imagine you don't want to have the control of which Redis instance is used-- you just want to enqueue and have the distribution be managed automatically. It looks like only one person has requested this so far , and they came up with a solution that uses Redis::Distributed :

datastore_config = YAML.load(ERB.new(File.read(File.join(Rails.root, "config", "redis.yml"))).result)

datastore_config = datastore_config["defaults"].merge(datastore_config[::Rails.env])

if datastore_config[:host].is_a?(Array)
  if datastore_config[:host].length == 1
    datastore_config[:host] = datastore_config[:host].first
  else
    datastore_config = datastore_config[:host].map do |host|
      host_has_port = host =~ /:\d+\z/

      if host_has_port
        "redis://#{host}/#{datastore_config[:db] || 0}"
      else
        "redis://#{host}:#{datastore_config[:port] || 6379}/#{datastore_config[:db] || 0}"
      end
    end
  end
end

Sidekiq.configure_server do |config|
  config.redis = ::ConnectionPool.new(:size => Sidekiq.options[:concurrency] + 2, :timeout => 2) do
    redis = if datastore_config.is_a? Array
      Redis::Distributed.new(datastore_config)
    else
      Redis.new(datastore_config)
    end

    Redis::Namespace.new('resque', :redis => redis)
  end
end

Another thing to consider in your quest to get high-availability and fail-over is to get Sidekiq Pro which includes reliability features: "The Sidekiq Pro client can withstand transient Redis outages. It will enqueue jobs locally upon error and attempt to deliver those jobs once connectivity is restored." Since sidekiq is for background processes anyway, a short delay if a Redis instance goes down should not affect your application. If one of your two Redis instances goes down and you're using round robin, you've still lost some jobs unless you're using this feature.

Answer 2

As carols10cents says its pretty simple but as I always like to encapsulate the capability and be able to reuse it in other projects I updated an idea from a blog from Hotel Tonight . This following solution improves upon Hotel Tonight's that does not survive Rails 4.1 & Spring preloader.

Currently I make do with adding the following files to lib/remote_sidekiq/ :

remote_sidekiq.rb

class RemoteSidekiq
  class_attribute :redis_pool
end

remote_sidekiq_worker.rb

require 'sidekiq'
require 'sidekiq/client'

module RemoteSidekiqWorker
  def client
    pool = RemoteSidekiq.redis_pool || Thread.current[:sidekiq_via_pool] || Sidekiq.redis_pool
    Sidekiq::Client.new(pool)
  end

  def push(worker_name, attrs = [], queue_name = "default")
    client.push('args' => attrs, 'class' => worker_name, 'queue' => queue_name)
  end
end

You need to create a initializer that sets redis_pool

config/initializers/remote_sidekiq.rb

url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

EDIT by Aleks:

In never versions of sidekiq, instead of lines:

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

use lines:

redis_remote_options = {
  namespace: "yournamespace",
  url: ENV.fetch("REDISCLOUD_URL")
}

RemoteSidekiq.redis_pool = Sidekiq::RedisConnection.create(redis_remote_options)

You can then simply the include RemoteSidekiqWorker module wherever you want. Job done!

**** FOR MORE LARGER ENVIRONMENTS ****

Adding in RemoteWorker Models adds extra benefits:

1. You can reuse the RemoteWorkers everywhere including the system that has access to the target sidekiq workers. This is transparent to the caller. To use the "RemoteWorkers" form within the target sidekiq system simply do not use an initializer as it will default to using the local Sidekiq client.
2. Using RemoteWorkers ensure correct arguments are always sent in (the code = documentation)
3. Scaling up by creating more complicated Sidekiq architectures is transparent to the caller.

Here is an example RemoteWorker

class RemoteTraceWorker
  include RemoteSidekiqWorker
  include ActiveModel::Model

  attr_accessor :message

  validates :message, presence: true

  def perform_async
    if valid?
      push(worker_name, worker_args)
    else
      raise ActiveModel::StrictValidationFailed, errors.full_messages
    end
  end

  private

  def worker_name
    :TraceWorker.to_s
  end

  def worker_args
    [message]
  end
end

Answer 3

I came across this and ran into some issues because I'm using ActiveJob , which complicates how messages are read out of the queue.

Building on ARO's answer, you will still need the redis_pool setup:

remote_sidekiq.rb

class RemoteSidekiq
  class_attribute :redis_pool
end

config/initializers/remote_sidekiq.rb

url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

Now instead of the worker we'll create an ActiveJob Adapter to queue the request:

lib/active_job/queue_adapters/remote_sidekiq_adapter.rb

require 'sidekiq'

module ActiveJob
  module QueueAdapters
    class RemoteSidekiqAdapter
      def enqueue(job)
        #Sidekiq::Client does not support symbols as keys
        job.provider_job_id = client.push \
          "class"   => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
          "wrapped" => job.class.to_s,
          "queue"   => job.queue_name,
          "args"    => [ job.serialize ]
      end

      def enqueue_at(job, timestamp)
        job.provider_job_id = client.push \
          "class"   => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
          "wrapped" => job.class.to_s,
          "queue"   => job.queue_name,
          "args"    => [ job.serialize ],
          "at"      => timestamp
      end

      def client
        @client ||= ::Sidekiq::Client.new(RemoteSidekiq.redis_pool)
      end
    end
  end
end

I can use the adapter to queue the events now:

require 'active_job/queue_adapters/remote_sidekiq_adapter'

class RemoteJob < ActiveJob::Base
  self.queue_adapter = :remote_sidekiq

  queue_as :default

  def perform(_event_name, _data)
    fail "
      This job should not run here; intended to hook into
      ActiveJob and run in another system
    "
  end
end

I can now queue the job using the normal ActiveJob api. Whatever app reads this out of the queue will need to have a matching RemoteJob available to perform the action.