[英]How to create a "buffer" program to act as a holding ground for MySql data lines, that will then be passed on when the cloud DB is available
I want to create a Python3 program that takes in MySQL data and holds it temporarily , and can then pass this data onto a cloud MySQL database.我想创建一个Python3程序来接收MySQL数据并临时保存它,然后可以将这些数据传递到云 MySQL 数据库中。
The idea would be that it acts as a buffer for entries in the event that my local network goes down , the buffer would then be able to pass those entries on at a later date, theoretically providing fault-tolerance .这个想法是,如果我的本地网络出现故障,它充当条目的缓冲区,然后缓冲区将能够在以后传递这些条目,理论上提供容错。
I have done some research into Replication and GTIDs and I'm currently in the process of learning these concepts.我对复制和 GTID 进行了一些研究,目前正在学习这些概念。 However I would like to write my own solution, or at least have it be a smaller program rather than a full implementation of replication server-side .
但是,我想编写自己的解决方案,或者至少让它成为一个较小的程序,而不是复制服务器端的完整实现。
I already have a program that generates some MySQL data to fill my DB, the key part I need help with would be the buffer aspect/implementation (The code itself I have isn't important as I can rework it later on).我已经有一个程序可以生成一些 MySQL 数据来填充我的数据库,我需要帮助的关键部分是缓冲区方面/实现(我拥有的代码本身并不重要,因为我可以稍后再修改它)。
I would greatly appreciate any good resources or help, thank you!我将不胜感激任何好的资源或帮助,谢谢!
I would implement what you describe using a message queue.我将使用消息队列实现您所描述的内容。
Example: https://hevodata.com/learn/python-message-queue/示例: https ://hevodata.com/learn/python-message-queue/
The idea is to run a message queue service on your local computer.这个想法是在本地计算机上运行消息队列服务。 Your Python application pushes items into the MQ instead of committing directly to the database.
您的 Python 应用程序将项目推送到 MQ,而不是直接提交到数据库。
Then you need another background task, called a worker, which you may also write in Python or another language, which consumes items from the MQ and writes them to the cloud database when it's available.然后,您需要另一个后台任务,称为 worker,您也可以使用 Python 或其他语言编写它,它使用 MQ 中的项目并在可用时将它们写入云数据库。 If the cloud database is not available, then the background worker pauses.
如果云数据库不可用,则后台工作人员会暂停。
The data in the MQ can grow while the background worker is paused. MQ 中的数据可以在后台工作程序暂停时增长。 If this goes on too long, you may run out of space.
如果持续时间过长,您可能会用完空间。 But hopefully the rate of growth is slow enough and the cloud database is available regularly, so the risk of this happening is low.
但希望增长速度足够慢,并且云数据库定期可用,因此发生这种情况的风险很低。
Re your comment about performance.重新评论您对性能的评论。
This is a different application architecture, so there are pros and cons.这是一个不同的应用程序架构,因此有利有弊。
On the one hand, if your application is "writing" to a local MQ instead of the remote database, it's likely to appear to the app as if writes have lower latency.一方面,如果您的应用程序正在“写入”到本地 MQ 而不是远程数据库,则应用程序可能会认为写入具有较低的延迟。
On the other hand, posting to the MQ does not write to the database immediately.另一方面,发布到 MQ 不会立即写入数据库。 There still needs to be a step of the worker pulling an item and initiating its own write to the database.
仍然需要工作人员拉出一个项目并启动它自己对数据库的写入的步骤。 So from the application's point of view, there is a brief delay before the data appears in the database, even when the database seems available.
所以从应用程序的角度来看,在数据出现在数据库中之前会有一个短暂的延迟,即使数据库似乎可用。
So the app can't depend on the data being ready to be queried immediately after the app pushes it to the MQ.所以应用程序不能依赖于在应用程序将数据推送到 MQ 后立即准备好查询的数据。 That is, it might be pretty prompt, under 1 second, but that's not the same as writing to the database directly, which ensures that the data is ready to be queried immediately after the write.
也就是说,它可能非常迅速,不到 1 秒,但这与直接写入数据库不同,这样可以确保数据在写入后立即准备好进行查询。
The performance of the worker writing the item to the database should be identical to that of the app writing that same item to the same database.将项目写入数据库的工作人员的性能应该与将相同项目写入同一数据库的应用程序的性能相同。 From the database perspective, nothing has changed.
从数据库的角度来看,没有任何改变。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.