[英]Rails + Sidekiq Csv Import Error
I just switched my CSV upload process to run on a worker. 我刚刚将CSV上传过程切换为在工作程序上运行。 It works fine locally, but when I try to upload a file in production I'm getting this error.
它在本地工作正常,但是当我尝试在生产环境中上传文件时,出现此错误。 It seems to me that it just doesn't know where to grab the file from
在我看来,它只是不知道从哪里获取文件。
017-02-22T16:32:48.914560+00:00 app[worker.1]: 4 TID-os5wk7tgo InventoryUploadWorker JID-f5be1032c019c28684582427 INFO: start
2017-02-22T16:32:49.224819+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.53973862.c2c36482-5d99-4a68-a399-0918d1ed36d2 sample#load_avg_1m=0.29 sample#load_avg_5m=0.07 sample#load_avg_15m=0.02
2017-02-22T16:32:49.224900+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.53973862.c2c36482-5d99-4a68-a399-0918d1ed36d2 sample#memory_total=144.37MB sample#memory_rss=134.18MB sample#memory_cache=6.66MB sample#memory_swap=3.54MB sample#memory_pgpgin=55377pages sample#memory_pgpgout=19323pages sample#memory_quota=512.00MB
2017-02-22T16:32:49.167416+00:00 app[worker.1]: Company Load (0.6ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT 1 [["id", 32]]
2017-02-22T16:32:49.246868+00:00 app[worker.1]: 4 TID-os5wk7tgo InventoryUploadWorker JID-f5be1032c019c28684582427 INFO: fail: 0.332 sec
2017-02-22T16:32:49.247408+00:00 app[worker.1]: 4 TID-os5wk7tgo WARN: {"class":"InventoryUploadWorker","args":["/tmp/RackMultipart20170222-4-1jaehp1.csv","32"],"retry":false,"queue":"default","jid":"f5be1032c019c28684582427","created_at":1487781168.915459,"enqueued_at":1487781168.9161458}
2017-02-22T16:32:49.247452+00:00 app[worker.1]: 4 TID-os5wk7tgo WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - /tmp/RackMultipart20170222-4-1jaehp1.csv
Worker: 工人:
class InventoryUploadWorker
include Sidekiq::Worker
sidekiq_options retry: false
Sidekiq.configure_server do |config|
config.redis = { url: ENV["REDISTOGO_URL"], network_timeout: 5 }
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV["REDISTOGO_URL"], network_timeout: 5 }
end
def perform(file_path, company_id)
CsvImport.csv_import(file_path, Company.find(company_id))
end
end
Import Method: 导入方式:
class CsvImport
def self.csv_import(filename, company)
time = Benchmark.measure do
File.open(filename) do |file|
headers = file.first
file.lazy.each_slice(150) do |lines|
Part.transaction do
inventory = []
insert_to_parts_db = []
rows = CSV.parse(lines.join, write_headers: true, headers: headers)
rows.map do |row|
part_match = Part.find_by(part_num: row['part_num'])
new_part = build_new_part(row['part_num'], row['description']) unless part_match
quantity = row['quantity'].to_i
row.delete('quantity')
row["condition"] = match_condition(row)
quantity.times do
part = InventoryPart.new(
part_num: row["part_num"],
description: row["description"],
condition: row["condition"],
serial_num: row["serial_num"],
company_id: company.id,
part_id: part_match ? part_match.id : new_part.id
)
inventory << part
end
end
#activerecord-import (bulk import)
InventoryPart.import inventory
end
end
end
end
puts time
end
It's not a good idea for the sidekiq process to rely on a temporary file from the web process. 对于sidekiq进程来说,依靠Web进程中的临时文件不是一个好主意。 What happens if the job fails and retries for the next week?
如果工作失败并在下周重试,会发生什么? What happens if your web and worker processes are on different machines or in different containers?
如果您的Web和工作进程位于不同的计算机或不同的容器中,该怎么办?
You should push the CSV contents as an argument or move the file to a well-known spot for the worker to pick up. 您应该将CSV内容作为参数推送,或将文件移动到知名位置,以供工作人员提取。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.