EOFError: end of file reached error and then memory error on heroku with workers

Question

I have a worker on heroku (1x) that I use for a sidekiq background process. This process basically cycles through all users and sends them an sms message via twilio.

Everything worked fine for months now but the last two times my customer tried to use this feature, it failed. I'm not sure why it failed the first time, but the most recent time (today), the worker failed and said:

EOFError: end of file reached

then, when looking at heroku logs --tail, it kept saying this over and over again:

2014-07-18T01:06:59.544697+00:00 heroku[worker.1]: Process running mem=804M(157.2%)
2014-07-18T01:06:59.544945+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

restarting the server finally fixed it. I had this exact issue a while ago that I thought I fixed. Here's my sidekiq.rb file:

Sidekiq.configure_server do |config|
  pool_size = Sidekiq.options[:concurrency] + 2

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['pool'] = pool_size
    ActiveRecord::Base.establish_connection(config)
  end
end

why is this happening all of a sudden? their user base is growing for sure, but it's not THAT much more than when it was working all the other times. thanks!

Answer 1

正如您在其他问题中回答的那样，您可以使用find_each来确保没有加载太多，并将该数据馈送到librato中进行监视