Run python scripts concurrently embedded in C++

Question

Is it possible to run scripts ie. with PyRun_SimpleFile() in separate C++ threads so they run concurrently?

I suspect no because of the GIL. But if there is any kind of trick making it possible please tell.

Best Regards Jakob

Answer 1

You can run them concurrently, but the GIL prevents them from running in parallel and utilizing your multiple CPU cores fully. This can be good enough if you mostly dispatch to other long running C++/CUDA code that releases the GIL again and if you only have a few CPUs.

This would be the approach taken by Sub-Interpreters , multiple mostly independent python interpreters that share only some parts, but sadly this includes the GIL, so it brings no benefits for parallel processing, and only small benefits for isolation between concurrently running scrips. One example of this approach is mod_wsgi .

Sub interpreters are a rarely used feature, so expect weird bugs/race conditions in C extensions that don't expect them.

If you said C# or Java instead of C++, you could use IronPython or Jython, to achieve your goal, as those do not have a GIL.

With CPython you can only do safely what multiprocessing does. The multiprocessing modul sidesteps the design issues of the GIL and just starts an extra process, but if you have expensive shared memory structures and no fork() syscall, this fails to deliver.

Of course, if you start to hack on the python implementation itself, you might be able to isolate all global state into a smaller contexts. But if you do that, you might as well just use a language that is designed for this multi-threaded embedding usecase like Tcl (look at NaviServer or F5s irules ) or Lua .