Redis 在弹性豆茎中运行 sidekiq 时出现连接超时错误

Question

I'm trying to set up sidekiq to connect to redis elasticache (in-transit encryption enabled) from my AWS elastic beanstalk (Amazon Linux 2) instance. I've followed links such as this , but am getting a 'Redis::TimeoutError: Connection timed out' when I do so. So now instead of deploying the scripts that run the 'bundle exec sidekiq -e production' command, I am just running that command in the /var/app/current directory of my ec2 instance with root privileges. Unfortunately that error is still appearing, and I am unsure why. My security groups are all configured correctly, and I can confirm the endpoint is correct.

The weird part is, I can connect to my redis endpoint on elasticache by running this command: openssl s_client -quiet -connect my_redis_endpoint:6379 and it works correctly. But sidekiq will fire up, then time out when I run that. Here is that output:

---

[root@ip-10-0-2-138 current]# bundle exec sidekiq -e production 2020-12-04T05:30:39.930Z pid=2910 tid=2xygu INFO: Booting Sidekiq 6.0.0 with redis options {:url=>"redis://master.redisproductionelasticache.bmxvqz.use2.cache.amazonaws.com:6379/12", :ssl=>true, :network_timeout=>5, :id=>"Sidekiq-server-PID-2910"}

2020-12-04T05:31:04.271Z pid=2910 tid=1e7u1a class=MessageBroadcastJob jid=82dd4217186de283e9d9bb70 elapsed=10.251 INFO: fail

2020-12-04T06:10:30.391Z pid=2910 tid=1e7wm6 WARN: {"context":"Job raised exception","job":{"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"MessageBroadcastJob","queue":"default","args":[{"job_class":"MessageBroadcastJob","job_id":"4d7d5195-7444-44c9-81c7-7f9a0c6b1f75","provider_job_id":null,"queue_name":"default","priority":null,"arguments":[{"_aj_globalid":"gid://e-wagers/Message/57bed09a-ef2d-4963-8288-65fe733347ba"}],"executions":0,"locale":"en"}],"retry":true,"jid":"a8f258e1feeecbf8c40a21a5","created_at":1607059284.1670992,"enqueued_at":1607062220.2854364,"error_message":"Connection timed out","error_class":"Redis::TimeoutError","failed_at":1607059284.1989071,"retry_count":6,"retried_at":1607060724.9108222},"jobstr":"

---

Here are my configuration files:

# config/initializers/sidekiq.rb

require 'sidekiq'
require 'sidekiq/web'

rails_root = Rails.root || File.dirname(__FILE__) + '/../..'
rails_env = Rails.env || 'development'
redis_config = YAML.load_file(rails_root.to_s + '/config/redis.yml')
redis_config.merge! redis_config.fetch(Rails.env, {})
redis_config.symbolize_keys!

Sidekiq.configure_server do |config|
 config.redis = { url: "redis://#{redis_config[:host]}:#{redis_config[:port]}/12", ssl: true, network_timeout: 5 }
end

Sidekiq.configure_client do |config|
 config.redis = { url: "redis://#{redis_config[:host]}:#{redis_config[:port]}/12", ssl: true, network_timeout: 5 }
end

development:
  host: 'localhost'
  port: 6379
  db: 5
production:
  host: 'master.my_redis_endpoint.bmxvqz.use2.cache.amazonaws.com'
  port: 6379
  db: 5

# config/sidekiq.yml

:concurrency: 25
:queues:
  - [mailers, 3]
  - ["priority", 2]
  - ["default", 1]

Any ideas what could be causing this? Unfortunately there are not a lot of resources online concerning sidekiq with elasticache, elastic beanstalk, and the newer Amazon Linux 2 platform.

Answer 1

Have you tried to use rediss:// instead of redis://

Answer 2

Since your Sidekiq process is receiving a job from the server, it is able to connect to Redis. What you're probably seeing is another gem's connection to Redis failing due to an overly-aggressive timeout. In my case, that gem was Redlock, which was setting a 100ms timeout (a little too slow for the Elasticache connection, which had ~120ms latency): https://github.com/leandromoreira/redlock-rb/blob/40756981932ca53c3e32bf3c68aa5ea9b772ef16/lib/redlock/client.rb#L9