[英]Passing variables between two python processes
I am intended to make a program structure like below 我打算制作一个如下的程序结构
PS1 is a python program persistently running. PS1是一个持续运行的python程序。 PC1, PC2, PC3 are client python programs.
PC1,PC2,PC3是客户端python程序。 PS1 has a variable hashtable, whenever PC1, PC2... asks for the hashtable the PS1 will pass it to them.
PS1具有变量哈希表,每当PC1,PC2 ...要求哈希表时,PS1会将其传递给它们。
The intention is to keep the table in memory since it is a huge variable (takes 10G memory) and it is expensive to calculate it every time. 这样做的目的是将表保留在内存中,因为它是一个巨大的变量(占用10G内存),并且每次计算都非常昂贵。 It is not feasible to store it in the hard disk (using pickle or json) and read it every time when it is needed.
将其存储在硬盘中(使用pickle或json)并在需要时每次读取它都是不可行的。 The read just takes too long.
读取时间太长。
So I was wondering if there is a way to keep a python variable persistently in the memory , so it can be used very fast whenever it is needed. 所以我想知道是否有一种方法可以将python变量永久保存在内存中 ,以便可以在需要时快速使用它。
You are trying to reinvent a square wheel, when nice round wheels already exist! 当好的轮毂已经存在时,您正在尝试重塑方形轮毂!
Let's go one level up to how you have described your needs: 让我们上一层描述您的需求:
IMHO, we are exactly facing what databases were created for. 恕我直言,我们正面临着创建数据库的目的。 For common use cases, having many processes all using their own copy of a 10G object is a memory waste, and the common way is that one single process have the data, and the others send requests for the data.
对于常见用例,让许多进程全部使用其自己的10G对象副本是一种内存浪费,并且常见的方式是一个进程拥有数据,而其他进程发送对数据的请求。 You did not describe your problem enough, so I cannot say if the best solution will be:
您对问题的描述不够,所以我不能说最好的解决方案是:
TL/DR: My advice is to experiment what are the performances with a good quality database and optionaly a dedicated chache. TL / DR:我的建议是尝试使用高质量的数据库和可选的专用cheche来测试性能。 Those solution allow almost out of the box load balancing on different machines.
这些解决方案几乎可以在不同机器上实现开箱即用的负载平衡。 Only if that does not work carefully analyze the memory requirements and be sure to document the limits in number of client processes and database size for future maintenance and use shared memory - read-only data being an hint that shared memory can be a nice solution
只有在这种方法不起作用的情况下,才能仔细分析内存需求,并确保记录客户端进程数和数据库大小的限制,以便将来维护和使用共享内存-只读数据暗示共享内存可以是一个不错的解决方案
In short, to accomplish what you are asking about, you need to create a byte array as a RawArray from the multiprocessing.sharedctypes module that is large enough for your entire hashtable in the PS1 server, and then store the hashtable in that RawArray. 简而言之,要实现您的要求,您需要从multiprocessing.sharedctypes模块创建一个字节数组作为RawArray,该字节数组对于PS1服务器中的整个哈希表足够大,然后将哈希表存储在该RawArray中。 PS1 needs to be the process that launches PC1, PC2, etc., which can then inherit access to the RawArray.
PS1需要是启动PC1,PC2等的进程,然后可以继承对RawArray的访问。 You can create your own class of object that provides the hashtable interface through which the individual variables in the table are accessed that can be separately passed to each of the PC# processes that reads from the shared RawArray.
您可以创建自己的对象类,该对象类提供哈希表接口,通过该接口可以访问表中的各个变量,这些变量可以分别传递给从共享RawArray读取的每个PC#进程。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.