Reconnect with Redis after Puma fork

Question

I am using a global variable in a rails application to store a redis client using the redis gem . In a config/initializers/redis.rb , I have

$redis = Redis.new(host: "localhost", port: 6379)

Then in application code, I use $redis to work the data in the Redis store.

I also use puma as the web server in production environment, and capistrano to deploy code. In the deploy process, capistrano restarts puma.

Every time I start or restart the puma web servers, I always get an "Internal Server Error" when I first use $redis to access data in the Redis store. I saw errors like Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.)

Searching around with google and stackoverflow led me to think that I needed to reconnect to Redis after puma forks child processes. So, I added in my config/puma.rb :

on_worker_boot do
  $redis.ping
end

But I was still getting the "Internal Server Error" caused by Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.) .

I saw this post http://qiita.com/yaotti/items/18433802bf1720fc0c53 . I then tried adding in config/puma.rb :

on_restart do
  $redis.quit
end

That did not work.

I tried in config/initializers/redis.rb to $redis.ping right after Redis.new . That did not work either.

I got this error if puma was started with no puma processes running, or restarted when an instance of puma process was running.

Refreshing the page would get me past this error. But I want to get rid of this even on the first attempt to use $redis . I was thinking that I did not use the redis gem or configure its reconnection correctly. Could someone tell me:

1. Is that the right way to use redis gem in a rails application?
2. How should the redis connection be reconnected in puma ?

puma gem documentation says, "You should place code to close global log files, redis connections, etc in this block so that their file descriptors don't leak into the restarted process. Failure to do so will result in slowly running out of descriptors and eventually obscure crashes as the server is restart many times." It was talking about the on_restart block. But it did not say how that should be done.

Answer 1

I was able to fix the error with a monkeypatch. This changes the behaviour so it just reconnects instead of throwing the Redis::InheritedError

###### MONKEYPATCH redis-rb 
# https://github.com/redis/redis-rb/issues/364
# taken from https://github.com/redis/redis-rb/pull/389/files#diff-597c124889a64c18744b52ef9687c572R314
class Redis
  class Client
   def ensure_connected
      tries = 0

      begin
        if connected?
          if Process.pid != @pid
            reconnect
          end
        else
          connect
        end

        tries += 1

        yield
      rescue ConnectionError
        disconnect

        if tries < 2 && @reconnect
          retry
        else
          raise
        end
      rescue Exception
        disconnect
        raise
      end
    end
  end
end
## MONKEYPATCH end

Answer 2

I'm running a Rails Application with IdentityCache using Puma in clustered mode with workers=4.

It is essential that the reconnects happen in the on_worker_boot callback.

I have to reconnect both the Rails.cache and the IdentityCache to avoid restart errors. Here's what I got working:

puma-config.rb

on_worker_boot do
   puts 'On worker boot...'
   puts "Reconnecting Rails.cache"
   Rails.cache.reconnect
   begin
      puts "Reconnecting IdentityCache"
      IdentityCache.cache.cache_backend.reconnect
   rescue Exception => e
      puts "Error trying to reconnect identity_cache_store: #{e.message}"
   end
end

Then I see the following in my logs, showing me the proof that it all works.

On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
[7109] - Worker 7115 booted, phase: 0
[7109] - Worker 7123 booted, phase: 0
[7109] - Worker 7119 booted, phase: 0
[7109] - Worker 7127 booted, phase: 0

Sure enough, the first request problems that used to be there after server restart are gone. QED.

Answer 3

Here's what I did:

  Redis.current.client.reconnect
  $redis = Redis.current

($redis is my global instance of a redis client)

Answer 4

I've put this into my config/puma.rb file, works for me.

on_restart do
  $redis = DiscourseRedis.new
  Discourse::Application.config.cache_store.reconnect
end
on_worker_boot do
  $redis = DiscourseRedis.new
  Discourse::Application.config.cache_store.reconnect
end

Answer 5

1. Upgrade redis-rb to 3.1.0 or above. The detail https://github.com/redis/redis-rb/pull/414/files#
2. monkey patch

# https://github.com/redis/redis-rb/pull/414/files#diff-5bc007010e6c2e0aa70b64d6f87985c20986ee1b2882b63a89b52659ee9c91f8
class Redis
  class Client
    def ensure_connected
      tries = 0
      begin
        if connected?
          if Process.pid != @pid
            raise InheritedError,
              "Tried to use a connection from a child process without reconnecting. " +
              "You need to reconnect to Redis after forking."
          end
        else
          connect
        end
        tries += 1

        yield
      rescue ConnectionError, InheritedError
        disconnect

        if tries < 2 && @reconnect
          retry
        else
          raise
        end
      rescue Exception
        disconnect
        raise
      end
    end
  end
end