Even though the first example about the ThreadPoolExecutor uses the pow function ( https://docs.python.org/3/library/concurrent.futures.html ) it seems that using pow abnormally blocks all threads when called. See code below.
from concurrent.futures import ThreadPoolExecutor
import time
executor = ThreadPoolExecutor(max_workers=16)
def blocking():
i = 0
while True:
time.sleep(1)
i+=1
print("block",i)
if i>3:
print("Starting pow....")
break
block= pow(363,100000000000000)
return True
def non_blocking():
i = 0
while True:
time.sleep(1)
i+=1
print("non",i)
return True
f1 = executor.submit(blocking)
f2 = executor.submit(non_blocking)
I expected the output:
block 1
non 1
block 2
non 2
block 3
non 3
block 4
Starting pow....
non 4
non 5
non 6
non 7
non 8
but the program stops running after "starting pow...." giving the result:
block 1
non 1
block 2
non 2
block 3
non 3
block 4
Starting pow....
Your python implementation (probably CPython) must have a Global Interpreter Lock (GIL)
pow
is a native Python function, which doesn't release the GIL when called, efficiently blocking all executions while it's running and it's running quite a while.
If you want non-blocking, use ProcessPoolExecutor
instead. A separate process means 2 separate GILs and not blocking. It has the exact same interface. It also has more constraints as it requires parameters to be picklable (no convenient shared memory like with threads)
Note that this computation probably never ends or ends up with "out of memory error": since it's power between integers with a big power value, so it doesn't overflow like floats would do but computes and computes, creating bigger and bigger integers each time
Estimated numbers of digits for this number is roughly:
>>> math.log10(363)*100000000000000
255990662503611.25
10^14 digits that is more than any current computer can handle memory wise, and also CPU-wise.
This is because of Python's global interpreter lock (GIL).
If you look at the bytecode for your function:
import time
def blocking():
i = 0
while True:
time.sleep(1)
i+=1
print("block",i)
if i>3:
print("Starting pow....")
break
block= pow(363,100000000000000)
return True
import dis
dis.dis(blocking)
which looks like this:
4 0 LOAD_CONST 1 (0)
2 STORE_FAST 0 (i)
5 4 SETUP_LOOP 50 (to 56)
6 >> 6 LOAD_GLOBAL 0 (time)
8 LOAD_METHOD 1 (sleep)
10 LOAD_CONST 2 (1)
12 CALL_METHOD 1
14 POP_TOP
7 16 LOAD_FAST 0 (i)
18 LOAD_CONST 2 (1)
20 INPLACE_ADD
22 STORE_FAST 0 (i)
8 24 LOAD_GLOBAL 2 (print)
26 LOAD_CONST 3 ('block')
28 LOAD_FAST 0 (i)
30 CALL_FUNCTION 2
32 POP_TOP
9 34 LOAD_FAST 0 (i)
36 LOAD_CONST 4 (3)
38 COMPARE_OP 4 (>)
40 POP_JUMP_IF_FALSE 6
10 42 LOAD_GLOBAL 2 (print)
44 LOAD_CONST 5 ('Starting pow....')
46 CALL_FUNCTION 1
48 POP_TOP
11 50 BREAK_LOOP
52 JUMP_ABSOLUTE 6
54 POP_BLOCK
12 >> 56 LOAD_GLOBAL 3 (pow)
58 LOAD_CONST 6 (363)
60 LOAD_CONST 7 (100000000000000)
62 CALL_FUNCTION 2
64 STORE_FAST 1 (block)
13 66 LOAD_CONST 8 (True)
68 RETURN_VALUE
You'll notice that the pow
call happens at 62 as a single bytecode instruction. The GIL can only be handed off between bytecodes, so since pow(363,100000000000000)
takes a long time to execute, the other thread doesn't get a chance to run during it.
pow(363,100000000000000)
if a big computation. It takes times for it to process and give the result
simple computation:
>>> import time
>>> def test(power):
... start = time.time()
... pow(363, power)
... return time.time() - start
...
>>> for power in range(1,11):
... print(test(pow(10, power)))
...
6.198883056640625e-06
2.1457672119140625e-06
3.0517578125e-05
0.0009307861328125
0.0421295166015625
1.7541508674621582 #This is a (363, 1000000)
Time increases exponentially
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.