[英]Rake Task timing out on Heroku - Move Rake task to worker using Sidekiq
I recently realized that my rake tasks can time out on Heroku, and I happen to have...a lot of them. 我最近意识到,我的耙子任务可能会在Heroku上超时,而我碰巧有很多。
Here is an example of my rake task: 这是我的瑞克任务的一个示例:
namespace :test do
desc "Test Rake Task"
task get_list: :environment do
require 'net/ftp'
sftp = Net::SFTP.start('ftp.test.com','luigi', :password => 'pass_word')
records = sftp.download!("luigi/list.xml")
records_hash = Hash.from_xml(records)
records_hash['Report']['Details'].each do |record|
contact = Contact.create(
first_name: record['FirstName'],
last_name: record['LastName'],
date_of_birth: record['DateofBirth']
)
if contact.valid?
puts "Created contact"
else
puts "Invalid contact"
end
end
end
end
I think that I need to move this to a background worker. 我认为我需要将其移交给后台工作者。 It could take 5+ minutes to loop through this task and add each contact to the database.
完成此任务并将每个联系人添加到数据库可能需要5分钟以上的时间。 I would like to use
Sidekiq
, but I've never used "workers" or set this up before. 我想使用
Sidekiq
,但是我从未使用过“ workers”或进行过设置。
In general, how would I go about setting Sidekiq
up to run on Heroku, and then move the above task to a background worker? 通常,我将如何设置
Sidekiq
使其在Heroku上运行,然后将上述任务移交给后台工作人员? I would like for this "worker" or "task" to be scheduled once a week, Monday mornings at 8 am. 我希望这个“工人”或“任务”计划每周一次,星期一上午8点安排。
My app settings: 我的应用程序设置:
Procfile:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
I am using a postgres Database, Rails 4.0.0, ruby 2.0.0, and as I mentioned earlier my app is hosted on Heroku. 我正在使用Postgres数据库,Rails 4.0.0,ruby 2.0.0,并且正如我之前提到的,我的应用程序托管在Heroku上。
You should follow the Sidekiq documentation and get it setup for Heroku. 您应该遵循Sidekiq文档并为Heroku进行设置。
One you have Sidekiq running, I recommend the following architecture: 您正在运行Sidekiq的一种,我建议使用以下体系结构:
ContactListWorker
: downloads list, enqueues a contact update job for each ContactListWorker
:下载列表,为每个列表排队一个联系人更新作业 ContactWorker
: given details, create/update a contact ContactWorker
:提供详细信息,创建/更新联系人 contacts:nightly_update
to queue up ContactListWorker job contacts:nightly_update
将ContactListWorker作业排队 Here's some rough pseudo code to show what that architecture might look like: 这是一些粗略的伪代码,以显示该体系结构可能是什么样的:
# rake contact:nightly_sync_list
namespace :contacts do
desc "Test Rake Task"
nightly_list_sync: :environment do
ContactListWorker.perform_async
end
end
class ContactListWorker
require 'net/ftp'
include Sidekiq::Worker
def perform()
sftp = Net::SFTP.start('ftp.test.com','luigi', :password => 'pass_word')
records = sftp.download!("luigi/list.xml")
records_hash = Hash.from_xml(records)
records_hash['Report']['Details'].each {|record| ContactWorker.perform_async(record) }
end
end
class ContactWorker
include Sidekiq::Worker
def perform(record)
contact = Contact.create(
first_name: record['FirstName'],
last_name: record['LastName'],
date_of_birth: record['DateofBirth']
)
if contact.valid?
puts "Created contact"
else
puts "Invalid contact"
end
end
end 结束
This architecture allows you to asynchronously kick off one background ContactListWorker
job. 这种体系结构允许您异步启动一个后台
ContactListWorker
作业。 This job will do the download and quickly enqueue N ContactWorker
jobs asynchronously. 该作业将完成下载并迅速使N
ContactWorker
作业ContactWorker
队。 This allows you to fan-out the processing across multiple Sidekiq workers and threads to distribute the processing. 这使您可以在多个Sidekiq工作器和线程之间展开处理,以分发处理。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.