简体   繁体   English

如何在不同conda环境的两个不同python进程之间交换数据?

[英]How to exchange data between two different python processes of different conda environments?

Two python scripts, A and B, have compatibility issues and need separate conda environments for them.两个 python 脚本 A 和 B 存在兼容性问题,需要单独的 conda 环境。 Here's the scenario.这是场景。 When script A runs, it sends data to process B (Script B is running in a different terminal), and process B returns the output to process A (Process A cannot be put to sleep).当脚本A运行时,它向进程B发送数据(脚本B在不同的终端运行),进程B返回output给进程A(进程A不能休眠)。 I have been using pickle files for exchanging data between these two processes, but this method seems slow, and I would like to speed it up, which is necessary for my work.我一直在使用pickle文件在这两个进程之间交换数据,但是这种方法看起来很慢,我想加快速度,这对我的工作来说是必要的。

  1. make one program a child of the other using the subprocess module and have the communication over stdin and stdout.使用subprocess模块使一个程序成为另一个程序的子程序,并通过标准输入和标准输出进行通信。 (fastest) (note you have to activate the other anaconda environment in the command to launch the child) (最快)(注意您必须在命令中激活其他 anaconda 环境才能启动子进程)
  2. have one application be a server and attach to a socket on localhost, the other application is going to be the client using the socket module.让一个应用程序成为服务器并附加到本地主机上的套接字,另一个应用程序将成为使用套接字模块的客户端。 (most organized and scalable solution) (最有组织和可扩展的解决方案)
  3. make a part of the memory a shared memory that both applications can access and write and read from using multiprocessing.shared_memory (requires proper synchronization, but can be faster than first option for transferring GBs of data at a time), (wrapping it in an io.TextIOWrapper will make communication a lot easier, as easy as working with sockets)使 memory 的一部分成为共享 memory 两个应用程序都可以使用multiprocessing.shared_memory访问和写入和读取(需要适当的同步,但可以比一次传输 GB 数据的第一个选项更快),(将其包装在io.TextIOWrapper将使通信更容易,就像使用套接字一样容易)

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM