Python: Using modules which support different Python versions in one project

Question

I have 2 python modules where one only supports Python 2.x and the other one 3.x. Unfortunately I need both for a project. My workaround for now is to have them run on their own as separate programs and building up their communication via the socket module.

1. I will be ending up with 2 executables, what I would like to avoid.
2. The "connection" between both modules has to be as fast as possible.

So my question is if there is a way to somehow combine both to one executable at the end and if there is a better solution for a fast communication as the client-server construction I have now.

Answer 1

There really is no good way to avoid that workaround.

Conceptually, there's no reason that you couldn't embed two interpreters into the same process. But practically, the CPython interpreter depends on some static/global state. While 3.7 is much better about that than, say, 3.0 or 2.6 was, that state still hasn't nearly been eliminated. ¹ And, the way C linkage works, there's no way to get around that without changing the interpreter.

Also, embedding CPython isn't hard , but it's not trivial , in the way that running an interpreter as a subprocess is trivial—and it may be harder than coming up with an efficient way to pass or share state between subprocesses.

Of course there are other interpreters besides CPython. But the other major implementation with both 2.7 and 3.x versions isn't easily embeddable (PyPy), and the two that are easily embeddable don't have 3.x versions, and also can only be embedded in another VM, and can't run C extension modules (Jython and IronPython). It is possible to do something like using JEP to embed CPython 3.7 via JNI in a JVM while also using Jython 2.7 natively in that same JVM, but I doubt that approach will work for you.

Meanwhile, I mentioned that passing or sharing data between processes generally isn't that hard.

• If you don't have that much data, you can usually just pass it pickled over a pipe.
• If you do have a ton of data, it usually is, or could be, stored in memory in some structured form—numpy arrays, big hunks of ASCII or UTF-8 text, arrays of ctypes structs, etc.—that you can overlay on an mmap or shared memory segment.
• Or, of course, you can come up with your own protocol and communicate with it over a (UNIX or IP) socket. But you don't necessarily have to jump right to that option.

Notice that multiprocessing supports both of the first two—although to take advantage of it with independent interpreters, you have to dig into its source and pull out the bits you need. And there are also third-party libraries that can help. (For example, if you need to pickle things that don't pickle natively, the answer is often as simple as "replace pickle with dill ".)

---

_{1. Running multiple subinterpreters in various restricted ways does sort of work with things like mod_wsgi , and PEP 554 aims to get things to the state where you can easily and cleanly run multiple 3.7 subinterpreters in the same process, but still nothing like completely independent embeddings of CPython—the subinterpreters share a GIL, a cycle collector, an atexit handler, etc.}