简体繁体 English

Message Queue VS Task Queue 架构

[英]Message Queue VS Task Queue architecture

原文 2023-02-01 10:18:18 6 1 design-patterns/ architecture/ message-queue/ task-queue

I want to build a product the can perform some Inte.net scans (in Python) to collect various kinds of data.我想构建一个可以执行一些 Inte.net 扫描（在 Python 中）以收集各种数据的产品。

I want to design it with tasks that perform these collecting jobs.我想用执行这些收集工作的任务来设计它。

There can be multiple scans that run parallelly on different inputs, so tasks can be duplicated, since they have different inputs to operate on.可以有多个扫描在不同的输入上并行运行，因此任务可以重复，因为它们有不同的输入要操作。

I wonder which architecture would fit for it, what technologies are the best.我想知道哪种架构适合它，哪种技术最好。

I thought of using RabbitMQ to store the tasks and Redis to store inputs.我想到了使用 RabbitMQ 来存储任务和 Redis 来存储输入。

The initial inputs trigger the scan, then each task spits his output that might be the input for other tasks.初始输入触发扫描，然后每个任务吐出他的 output 可能是其他任务的输入。

What do you think of this possible design?您如何看待这种可能的设计？ Can it be improved?可以改进吗？ Other technologies?其他技术？

1 个解决方案

It depends on the size of the inputs.这取决于输入的大小。 If those are relatively small I would go with just message broker and sending everything in the message (ie the task type and it's inputs) - otherwise some outside store is better used.如果这些相对较小，我会使用消息代理 go 并发送消息中的所有内容（即任务类型及其输入）——否则最好使用一些外部商店。 Depending on the durability requirements possibly a persistent storage (like database) should be considered.根据持久性要求，可能应考虑持久性存储（如数据库）。