[英]Node JS: Passing Response Object to Bull Queue for Server Side Events
I am stuck at an architectural decision.我被困在一个架构决定上。 I have Node + Express app, It has an API to upload files.
我有 Node + Express 应用程序,它有一个 API 来上传文件。 After the upload is done, the response is closed and uploaded file is processed by FFMPEG batch wise with the help of Bull Queue + Redis.
上传完成后,响应关闭,上传的文件由 FFMPEG 在 Bull Queue + Redis 的帮助下批量处理。 This structure works fine but recently I started testing on Server side events to give updates about processing to the end user.
这种结构工作正常,但最近我开始测试服务器端事件,以便向最终用户提供有关处理的更新。 But I am unable to pass the response object to Bull Queue to write regular updates from server to user.
但我无法将响应 object 传递给 Bull Queue,以便将服务器定期更新写入用户。
1. Imports 1. 进口
import childProcess from 'child_process';
import Bull from 'bull'
const Queue = new Bull('background_job', {redis: {port: process.env.port, host: process.env.host, password: process.env.password}});
2. Upload function 2.上传function
const uploadVideo = async(req, res) => {
try{
const result = await authUser(req);
const result2 = await checkUploadFile(result);
const result3 = await insertPost(result2, res);
await Queue.add(result3.data, result3.opts)
} catch(err){
res.status(403).send(err);
}
}
3. Promises 3. 承诺
const authUser = (req) => {
return new Promise((resolve, reject) => {
//do some work
})
}
const checkUploadFile = (result) => {
return new Promise((resolve, reject) => {
//do some more work
})
}
const insertPost= (result, res) => {
return new Promise((resolve, reject) => {
//do final work
...........
//preparing server side events
const headers = {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
'Access-Control-Allow-Origin': '*'
};
res.writeHead(200, headers);
res.write(JSON.stringify({status: true, id: 1})); //testing server side events for the first time
//Let's continue to Bull
const data = {res: res} <- error here: TypeError: Converting circular structure to JSON
const opts = {removeOnComplete: true, removeOnFail: true}
resolve({data: data, opts: opts});
})
}
4. Queue Process 4.排队过程
Queue.process((job, done) => {
const res = job.data.res
childProcess.execFile('someScript.sh', [`some`, `arguments`], { stdio: ['pipe', 'pipe', 'ignore']}, (err, stderr, stdout) => {
if(err){
done(new Error("Failed: " + err))
res.write(JSON.stringify({status: true, id: 2})); //here using SSE
res.end()
} else {
done()
res.write(JSON.stringify({status: false})); //here using SSE
res.end()
}
})
})
5. Error logged by PM2 5. PM2 记录的错误
TypeError: Converting circular structure to JSON
--> starting at object with constructor 'Socket'
| property 'parser' -> object with constructor 'HTTPParser'
--- property 'socket' closes the circle
I tried to use JSON.stringify(res)
to pass the response object as JSON but that didn't work either.我尝试使用
JSON.stringify(res)
将响应 object 作为 JSON 传递,但这也不起作用。 Now I'm considering if this approach is right or should I go with Socket.io (which is an overkill for a simple server side events)现在我正在考虑这种方法是否正确,或者我应该使用 go 和 Socket.io (这对于简单的服务器端事件来说太过分了)
Thank you谢谢
Why do you even write this line:你为什么还要写这一行:
const data = {res: res} <- error here: TypeError: Converting circular structure to JSON.
You still have access to the response object in uploadVideo function where you call insertPost.您仍然可以访问调用 insertPost 的 uploadVideo function 中的响应 object。 so it can simply be:
所以它可以简单地是:
await Queue.add(res, result3.opts).
For instance:例如:
const uploadVideo = async(req, res) => {
try{
const result = await authUser(req);
const result2 = await checkUploadFile(result);
const result3 = await insertPost(result2, res);
await Queue.add(res, result3.opts); // still have access to res
} catch(err){
res.status(403).send(err);
}
Remove this line:删除这一行:
const data = {res: res} <- error here: TypeError: Converting circular structure to JSON
Just use response只需使用响应
Queue.process((res, done) => {
//const res = job.data.res
childProcess.execFile('someScript.sh', [`some`, `arguments`], { stdio: ['pipe', 'pipe', 'ignore']}, (err, stderr, stdout) => {
if(err){
done(new Error("Failed: " + err))
res.write(JSON.stringify({status: true, id: 2})); //here using SSE
res.end()
} else {
done()
res.write(JSON.stringify({status: false})); //here using SSE
res.end()
}
})
});
Edit:编辑:
I see what you mean.我明白你的意思了。 Had a look at the bull module.
看了一下公牛模块。 Why cant you do something like this.
为什么你不能做这样的事情。
const uploadVideo = async(req, res) => {
try{
res.jobId = 0; // we need a way to know if job completed is our request const result = await authUser(req);
const result2 = await checkUploadFile(result);
const result3 = await insertPost(result2, res);
Queue.add({id: res.jobId, somedatafromresult3: 'result3.somedata' }, result3.opts);
Queue.on("completed", (err, data) => {
if (data.id === res.jobId) { // check to see if completed job is our one.
res.write(JSON.stringify(data)); //here using SSE
res.end()
}
console.log(data);
});
} catch(err){
res.status(403).send(err);
}
}
Then in your process function, simply return the data which will be emitted.然后在您的进程 function 中,只需返回将发出的数据。 ie
IE
videoQueue.process(function(job, done){
childProcess.execFile('someScript.sh', [`some`, `arguments`], { stdio: ['pipe', 'pipe', 'ignore']}, (err, stderr, stdout) => {
if(err){
done(err, {status: true, id: job.data.id});
} else {
done(null, {status: false, id: job.data.id});
}
})
})
; ;
You can use job.progress()
to communicate with the route that's connected to the client via SSE.您可以使用
job.progress()
与通过 SSE 连接到客户端的路由进行通信。 Update the progress with job.progress(percent)
, passing in a number.使用
job.progress(percent)
更新进度,传入一个数字。 The Express route scope can then spin on this and emit SSE events to the client as the job progresses. Express 路由 scope 然后可以在此基础上旋转,并随着作业的进行向客户端发送 SSE 事件。
Here's a basic runnable example as a proof of concept you can add your processing, error handling and job.progress
and SSE logic onto.这是一个基本的可运行示例作为概念证明,您可以将处理、错误处理以及
job.progress
和 SSE 逻辑添加到其中。
const express = require("express");
const fs = require("fs").promises;
const path = require("path");
const Queue = require("bull");
const sleep = (ms=1000) =>
new Promise(resolve => setTimeout(resolve, ms))
;
const queue = new Queue("test", process.env.REDIS_URL);
queue.process(4, async job => {
for (let i = 1; i <= job.data.seconds; i++) {
await job.progress(i / job.data.seconds * 100 | 0);
await sleep();
}
return Promise.resolve(`job ${job.id} complete!`);
});
const app = express();
app
.set("port", process.env.PORT || 5000)
.get("/", async (req, res) => {
try {
res.set({
"Access-Control-Allow-Origin": "*",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Content-Type": "text/event-stream",
});
res.flushHeaders();
const job = await queue.add({
seconds: Math.abs(+req.query.seconds) || 10,
});
let connected = true;
res.on("close", () => {
connected = false;
});
for (; connected; await sleep()) {
const j = await queue.getJob(job.id);
const progress = await j.progress();
res.write(`${progress}\n`);
if (progress >= 100) { // TODO handle job errors
break;
}
}
res.write(await job.finished());
}
catch (err) {
res.write(err.message);
}
finally {
res.end();
}
})
.listen(app.get("port"), () =>
console.log(`server listening on port ${app.get("port")}`)
)
;
Sample run:样品运行:
$ curl localhost:5000
0
10
20
30
40
50
60
70
80
90
100
job 64 complete!
See also How to use server-sent-events in express.js which has a sample client that can read the response stream.另请参阅如何在 express.js 中使用 server-sent-events,它有一个示例客户端,可以读取响应 stream。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.