[英]How to synchronize Node.js and Python when using RabbitMQ i.e run a task on a worker, wait for the result and send it to the client
I have an express Node.js Application and a machine learning algorithm using python.我有一个快速 Node.js 应用程序和一个使用 python 的机器学习算法。 I'm using RabbitMQ to integrate Node.Js and Python and it works, I mean, this is more performatic than using the library child_process with spawn, for example.
我正在使用 RabbitMQ 来集成 Node.Js 和 Python 并且它有效,我的意思是,这比使用带有 spawn 的库 child_process 更具性能,例如。 But I'm having trouble synchronizing workers' responses to their respective requests.
但是我在同步工人对他们各自请求的响应时遇到了麻烦。
My code is similar to the example below (toy example) and based on this post 1 .我的代码类似于下面的示例(玩具示例)并基于这篇文章1 。 This implementation has two main problems, it is sending the wrong answer to the client (postman) or is not completing the request.
这个实现有两个主要问题,它向客户端(邮递员)发送错误的答案或者没有完成请求。
This code should take a request from a client (image and type), put this task on the queue (tasks queue), waiting for the worker finish its job and send the result to the client (the right one).这段代码应该接受来自客户端的请求(图像和类型),将此任务放入队列(任务队列),等待工作人员完成其工作并将结果发送给客户端(右侧)。
import express from "express";
import bodyParser from 'body-parser';
import cors from 'cors';
import path from 'path';
import multer from "multer";
import amqp from "amqplib/callback_api.js";
const port = 3000;
const app = express();
app.use(express.json());
app.use(bodyParser.urlencoded({extended: true}));
app.use(cors());
const storage = multer.memoryStorage();
const upload = multer({ storage: storage });
app.get('/', (req, res) => {
res.status(200).send({message: "ok"});
});
app.post("/classify", upload.single("image"), async(req, res) => {
const { type } = req.body;
const task = Buffer.from(
JSON.stringify({
type: type,
image: req.file.buffer.toString("base64")
})
);
amqp.connect("amqp://localhost", (err, conn) => {
conn.createChannel((err, ch) => {
ch.assertQueue("tasks", {durable: false});
ch.assertQueue("results", {durable: false});
ch.sendToQueue("tasks", task);
ch.consume("results", (msg) => {
const predictions = JSON.parse(msg.content.toString());
res.status(200).send({message: predictions });
},{noAck: true});
});
setTimeout(() => {conn.close();}, 500)
});
});
console.clear();
app.listen(port, () => {
console.log(`Server listening at http://localhost:${port}`)
})
This code should take tasks from a queue (task queue), run them and put the result into the result queue.此代码应从队列(任务队列)中获取任务,运行它们并将结果放入结果队列。
#!/usr/bin/env python
import pika
import json
import time
import random
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.basic_qos(prefetch_count=1)
channel.queue_declare(queue="tasks")
channel.queue_declare(queue="results")
# Simulate the execution of a machine learning model
def run_model(image, type):
time.sleep(random.randint(1, 4))
return random.choice([" Iris setosa", "Iris virginica", "Iris versicolor"])
def callback(ch, method, properties, body):
params = json.loads(body.decode('utf-8'))
type = str(params["type"])
image = params['image']
print("Worker received a new task...")
results = run_model(image, type)
# send a message back
channel.basic_publish(
exchange="",
routing_key="results",
body=json.dumps(results, ensure_ascii=False)
)
#connection.close()
# receive message and complete the task
channel.basic_consume("tasks", callback, auto_ack=True)
channel.start_consuming()
How to solve these problems?如何解决这些问题?
I'm not super familiar with RabbitMQ, but I think you need a way to uniquely identify each task sent to the Python worker.我对 RabbitMQ 不是很熟悉,但我认为您需要一种方法来唯一标识发送给 Python 工作人员的每个任务。
Try generating a UUID on the Node.js side (use the uuid
package), and passing that along with the job data.尝试在 Node.js 端生成一个 UUID(使用
uuid
包),并将其与作业数据一起传递。 Then, consume the results-${uuid}
channel in Node.然后,使用 Node.js 中的
results-${uuid}
通道。
When the Python worker finishes, have it use the results-${uuid}
routing key from the passed-in params.当 Python worker 完成时,让它使用传入参数中的
results-${uuid}
路由键。 That way, each request is only listening for its particular result channel.这样,每个请求只监听其特定的结果通道。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.