使用并行性崩溃的程序：在没有 gil 的情况下不允许丢弃拥有的 Python 对象

Question

    %%cython
from threading import Thread
import time
def countdown(n):
    while n > 0:
        n -= 1

COUNT = 10000000


start = time.time()
t1 = Thread(target=countdown,args=(COUNT/2,))
t2 = Thread(target=countdown,args=(COUNT/2,))
with nogil:
    t1.start();t2.start()
    t1.join();t2.join()
    
end = time.time()
print(end-start)

I read on the Cython docs website that, almost every python code is workable in Cython. As we know that is a famous snippet to demonstrate the limitations of Python and GIL. I tried to recreate a solution in Cython.

However, I am getting this issue in compiling this code.

Error compiling Cython file:
------------------------------------------------------------
...

start = time.time()
t1 = Thread(target=countdown,args=(COUNT/2,))
t2 = Thread(target=countdown,args=(COUNT/2,))
with nogil:
    t1.start();t2.start()
           ^
------------------------------------------------------------

/Users/sanskar/.ipython/cython/_cython_magic_bb84bd2392afb02224d35c81782c007c.pyx:14:12: Discarding owned Python object not allowed without gil

Error compiling Cython file:
------------------------------------------------------------
...

I am fairly new to Cython, as evident from my code. I would like to have some suggestions on how would I fix it.

Answer 1

This is not going to work.

The threading library returns Python objects. To interact with them you need the GIL. You therefore cannot expect to call .start() or .join() without the GIL.

What's more useful is to wrap the parts of your function that don't need the GIL with a nogil block. For example the loop in countdown could easily by done without the GIL (there's actually a good chance that the C compiler will spot that this loop is completely pointless and remove it completely, but that's a separate issue)

def countdown(int n):
    with nogil:
        while n > 0:
            n -= 1

This allows the majority of the work to be done without the GIL, but requires the GIL to be held at the start and end of the function.

---

There's a secondary complication: regular Python code will release the GIL occasionally to allow other threads to run if needed. Cython typically won't (although it might invoke Python code that does, so you can't rely on this). Therefore it probably makes sense to write countdown in Cython but not the code that starts the threads.

I suspect this won't be an issue here but it's probably better to be safe.