Resque jobs can't connect to outside hosts

Question

In my Rails app, I've got ActiveJob setup to work with Resque. I've created the this job to send iOS push notifications with the grocer gem:

class SendPushNotificationJob < ActiveJob::Base
  queue_as :default

  def perform(device_token, msg)
    # TODO: Move the grocer initialization somewhere else.
    pusher = Grocer.pusher(
      certificate: ENV['PUSH_CERT_PATH'],
      passphrase:  ENV['PUSH_CERT_KEY'],
      gateway:     "gateway.sandbox.push.apple.com",
      port:        2195,
      retries:     3
    )

    Resque.logger.debug "PUSHER OBJECT IS #{pusher.inspect}!"

    notification = Grocer::Notification.new(
    device_token: device_token,
    alert: msg)

    Resque.logger.debug "NOTIFICATION OBJECT IS #{notification.inspect}"

    res = pusher.push(notification)

    Resque.logger.debug "PUSH RESULT IS #{res.inspect}"
  end
end

The output of the Resque log when this job is picked up is:

** [22:08:31 2016-07-06] 24074: Found job on default
** [22:08:31 2016-07-06] 24074: got: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"SendPushNotificationJob", "job_id"=>"c88619a4-d2ea-4b27-a3c8-3ea5be04a130", "queue_name"=>"default", "arguments"=>["<my_device_token>", "Test message"], "locale"=>"en"}])
** [22:08:31 2016-07-06] 24074: resque-1.26.0: Processing default since 1467860911 [ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper]
** [22:08:31 2016-07-06] 24074: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"SendPushNotificationJob", "job_id"=>"c88619a4-d2ea-4b27-a3c8-3ea5be04a130", "queue_name"=>"default", "arguments"=>["<my_device_token>", "Test message"], "locale"=>"en"}])]
** [22:08:31 2016-07-06] 24074: resque-1.26.0: Forked 24092 at 1467860911
** [22:08:31 2016-07-06] 24092: Running after_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"SendPushNotificationJob", "job_id"=>"c88619a4-d2ea-4b27-a3c8-3ea5be04a130", "queue_name"=>"default", "arguments"=>["<my_device_token>", "Test message"], "locale"=>"en"}])]
** [22:08:31 2016-07-06] 24092: PUSHER OBJECT IS #<Grocer::Pusher:0x007f862152f4f0 @connection=#<Grocer::Connection:0x007f862152f630 @certificate="<cert path>", @passphrase="<passphrase>", @gateway="gateway.sandbox.push.apple.com", @port=2195, @retries=3>>!
** [22:08:31 2016-07-06] 24092: NOTIFICATION OBJECT IS #<Grocer::Notification:0x007f8627a23f00 @identifier=0, @device_token="<my_device_token>", @alert="Test message", @encoded_payload=nil>

Then the exception happens:

** [22:08:31 2016-07-06] 24092: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"SendPushNotificationJob", "job_id"=>"c88619a4-d2ea-4b27-a3c8-3ea5be04a130", "queue_name"=>"default", "arguments"=>["<my_device_token>", "Test message"], "locale"=>"en"}]) failed: #<SocketError: getaddrinfo: nodename nor servname provided, or not known>

Here's what I know:

1. The gateway URL is being set properly.
2. The environment variables are correctly set.
3. This job works when using perform_now instead of perform_later .

Lastly, this is not an issue with grocer . I created another job that simply logs the source of a URL:

class TestExternalConnectionJob < ActiveJob::Base
  queue_as :default

  def perform(url)
    res = HTTParty.get(url)

    if res
      Resque.logger.debug "SUCCESS"
      Resque.logger.debug res.body
    else
      Resque.logger.debug "FAILURE"      
    end
  end
end

Running TestExternalConnectionJob.perform_later('http://stackoverflow.com') in the Rails console produces this output in the Resque log:

** [22:14:41 2016-07-06] 24285: Found job on default
** [22:14:41 2016-07-06] 24285: got: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"TestExternalConnectionJob", "job_id"=>"523da5ca-2b6c-452e-95ef-33edece69996", "queue_name"=>"default", "arguments"=>["http://stackoverflow.com"], "locale"=>"en"}])
** [22:14:41 2016-07-06] 24285: resque-1.26.0: Processing default since 1467861281 [ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper]
** [22:14:41 2016-07-06] 24285: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"TestExternalConnectionJob", "job_id"=>"523da5ca-2b6c-452e-95ef-33edece69996", "queue_name"=>"default", "arguments"=>["http://stackoverflow.com"], "locale"=>"en"}])]
** [22:14:41 2016-07-06] 24285: resque-1.26.0: Forked 24300 at 1467861281
** [22:14:41 2016-07-06] 24300: Running after_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"TestExternalConnectionJob", "job_id"=>"523da5ca-2b6c-452e-95ef-33edece69996", "queue_name"=>"default", "arguments"=>["http://stackoverflow.com"], "locale"=>"en"}])]
** [22:14:41 2016-07-06] 24300: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | [{"job_class"=>"TestExternalConnectionJob", "job_id"=>"523da5ca-2b6c-452e-95ef-33edece69996", "queue_name"=>"default", "arguments"=>["http://stackoverflow.com"], "locale"=>"en"}]) failed: #<SocketError: Failed to open TCP connection to stackoverflow.com:80 (getaddrinfo: nodename nor servname provided, or not known)>

So, tl;dr , none of my Resque jobs are able to connect to external services.

Appendix

My lib/resque.rake file:

require 'resque/tasks'

namespace :resque do
  task :setup => :environment do
    require 'resque'
  end
end

My config/initializers/resque.rb file:

require 'resque'

Resque.redis = Redis.new(url: 'redis://127.0.0.1:6379')
Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
Resque.logger.level = Logger::DEBUG
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }

Answer 1

Well, writing that question was an enormous waste of time.

For some reason, after a reboot and restarting all of my processes, I was able to connect to outside hosts.

Previously, I had restarted the processes many times without rebooting. I'm using eye to manage them, so several times in trying to solve this I issued an eye restart command to restart all of the processes my app depends on ( postgres , redis and resque-worker ). Every time after the restart I would get the same error.

But somehow, a reboot did the trick.

Computers are weird sometimes.