[英]NodeJS Bull Stop the queue jobs on a job failed
I have multiple Bull Queues in my NodeJS
project which will run if previous queue is executed successfully.我的
NodeJS
项目中有多个Bull队列,如果前一个队列成功执行,它们将运行。 I'm trying to verify some email addresses here.我试图在这里验证一些 email 地址。
Check the Email format (formatQueue)检查 Email 格式(formatQueue)
Email Existence using npm email-existence
package (existenceQueue) Email Existence using
npm email-existence
package (existenceQueue)
The formatQueue
is less time taking process, which wil run the RegEx
and validate the Email
format. formatQueue
是一个耗时的过程,它将运行RegEx
并验证Email
格式。 but The email-existence
package takes around 5-10 seconds to complete.但是
email-existence
package 大约需要 5-10 秒才能完成。
formatQueue
and existenceQueue
works properly if there are less jobs
like 20-100.如果像 20-100 这样的
jobs
较少, formatQueue
和existenceQueue
队列可以正常工作。 but when I Add more than that around 1000 jobs at a time, existenceQueue
failes with below error但是当我一次添加超过大约 1000 个作业时,
existenceQueue
失败并出现以下错误
myemail@email.com job stalled more than allowable limit
I checked the issue HERE and HERE , I thought the process is taking too long to respond, so added limiter
as refered HERE .我检查了HERE和HERE的问题,我认为该过程需要很长时间才能响应,因此添加了
limiter
,如HERE所述。 But that does not help me.但这对我没有帮助。
If a job in any of the queue fails, Its not processing the next job.如果任何队列中的作业失败,则不会处理下一个作业。 It will stop there and the other jobs will stay in
waiting
state.它将停在那里,其他作业将继续
waiting
state。
My code is something similar to below code.我的代码类似于下面的代码。 please help me with the issue.
请帮我解决这个问题。
Queue.js队列.js
var formatQueue = new Queue('format', "redis-db-url");
var existenceQueue = new Queue('existence', "redis-db-url");
// ------------ function for adding to queue ------------
module.exports.addToQueue = (emails) => {
emails.forEach(element => {
formatQueue.add(element, { attempts: 3, backoff: 1000 });
});
}
// ------------ Queue Process -------------
// Format Test Process
formatQueue.process(function(job, done){
FormatTest.validate(job.data, (err, data) => {
if(err) done();
else{
job.data = data;
done();
}
});
});
// Existence Test Process
formatQueue.process(function(job, done){
ExistenceTest.validate(job.data, (err, data) => {
if(err) done();
else{
job.data = data;
done();
}
});
});
// ------------ On Cmplete Handlers ------------
formatQueue.on('completed', function(job){
if(job.data.is_well_format){
existenceQueue.add(job.data, { attempts: 3, backoff: 1000 });
}else QueueModel.lastStep(job.data)
});
existenceQueue.on('completed', function(job){
QueueModel.lastStep(job.data)
});
// ------------ To update the emaile ------------
module.exports.lastStep = (data) => {
Emails.updateEmail(data, (err, updated) => {
if(!err) {
formatQueue.clean('completed');
existenceQueue.clean('completed');
}
})
}
--------- Update --------- - - - - - 更新 - - - - -
The processor was taking too much time to respond so the job was getting stalled
or getting failed since i was using timeout.处理器花费了太多时间来响应,因此由于我使用超时,作业被
stalled
或失败。
I'm trying to run the process
in different processor
file itsef as its in bull
documentation, I've added the file as below.我正在尝试在不同的
processor
文件 itef 中运行该process
,就像它在bull
文档中一样,我添加了如下文件。
// -------- Queue.js ----------
formatQueue.process(__dirname+"/processors/format-worker.js");
// On Cmplete Handler
formatQueue.on('completed', function(job, result){
console.log(result, "Format-Complete-job"); // result is undefined
if(job.data.is_well_format){
existenceQueue.add(job.data, { attempts: 3, backoff: 1000 });
}else QueueModel.lastStep(job.data)
});
// -------- Queue.js ends ---------
//format-worker.js
Validator = require("../../validators");
module.exports = (job) => {
Validator.Format.validate(job.data, (data) => {
job.data = data;
return Promise.resolve(data);
});
}
Now On Job complete which i was using before, I used to get job data with updated job parameters.现在在我之前使用的作业完成时,我曾经使用更新的作业参数获取作业数据。 Now I'm not getting updated job data.
现在我没有得到更新的工作数据。 and the second parameter which is there in the documentation ie
result
is undefined
.文档中的第二个参数,即
result
是undefined
。 Now how can I get the updated job data in this case.现在在这种情况下如何获取更新的作业数据。
Try repeatable jobs尝试可重复的工作
var formatQueue = new Queue('format', "redis-db-url");
var existenceQueue = new Queue('existence', "redis-db-url");
// ------------ function for adding to queue ------------
module.exports.addToQueue = (emails) => {
emails.forEach(element => {
let jobOptions = {
repeat: {
every: 10 * 1000, // Run job every 10 seconds for example
limit: 3 // Maximum number of times a job can repeat.
},
jobId: someUniqueId, // important do not forget this
removeOnComplete: true, // removes job from queue on success (if required)
removeOnFail: true // removes job from queue on failure (if required)
}
formatQueue.add(element, jobOptions);
});
}
// ------------ Queue Process -------------
// Format Test Process
formatQueue.process(function (job, done) {
FormatTest.validate(job.data, (err, data) => {
if (err) {
// Done with error
done(true);
} else {
job.data = data;
// Done without any error
done(false);
}
});
});
// Existence Test Process
existenceQueue.process(function (job, done) {
ExistenceTest.validate(job.data, (err, data) => {
if (err) {
// Done with error
done(true);
} else {
job.data = data;
// Done without any error
done(false);
}
});
});
// ------------ On Complete Handlers ------------
formatQueue.on('completed', function (job) {
if (job.data.is_well_format) {
let jobOptions = {
repeat: {
every: 10 * 1000, // Run job every 10 seconds for example
limit: 3 // Maximum number of times a job can repeat.
},
jobId: someUniqueId, // important do not forget this
removeOnComplete: true, // removes job from queue on success (if required)
removeOnFail: true // removes job from queue on failure (if required)
}
existenceQueue.add(job.data, jobOptions);
} else QueueModel.lastStep(job.data)
});
existenceQueue.on('completed', function (job) {
QueueModel.lastStep(job.data)
});
// ------------ To update the email ------------
module.exports.lastStep = (data) => {
Emails.updateEmail(data, (err, updated) => {
if (!err) {
formatQueue.clean('completed');
existenceQueue.clean('completed');
}
})
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.