将文件传输到消息代理
[英]Transferring files to a message broker
问题描述
We are setting up a database with a front end program that parses csv files and dumps them into the DB. 
我们正在使用一个前端程序来建立数据库，该程序可以解析csv文件并将其转储到DB中。 Files are coming from many locations at the same time and need to be persisted if anything goes wrong. 文件同时来自多个位置，如果出现任何问题，则需要保留这些文件。
Is a message broker like rabbitmq a good application for this? 像Rabbitmq这样的消息代理是否是一个很好的应用程序？ I need a queue of some sort that can be read from the back end. 我需要可以从后端读取的某种队列。
1 个解决方案
解决方案1
2 已采纳 2016-05-31 20:55:17
Is a message broker like rabbitmq a good application for this?
Yes, but I wouldn't recommend you to send large files to RabbitMQ especially if you'd like to use persistent messages. 是的，但是我不建议您将大文件发送到RabbitMQ，特别是如果您要使用持久性消息。 If this is the case, I'd save a file to a storage (cloud, cache, etc) and send its name/path in a message. 如果是这种情况，我会将文件保存到存储（云，缓存等）中，并在消息中发送其名称/路径。 A worker process would pick up a message, then download a file, parse it and save its contents to database. 辅助进程将接收一条消息，然后下载文件，对其进行解析并将其内容保存到数据库中。 This way you can distribute files among worker processes and reduce load on broker. 这样，您可以在辅助进程之间分发文件并减少代理的负载。 It would allow you to scale easily. 它可以让您轻松扩展。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.