Python：多线程不会改善运行时

Question

I'm trying to write simple multi-threaded python script:

from multiprocessing.dummy import Pool as ThreadPool

def resize_img_folder_multithreaded(img_fldr_src,img_fldr_dst,max_num_of_thread):

    images = glob.glob(img_fldr_src+'/*.'+img_file_extension)
    pool = ThreadPool(max_num_of_thread) 

    pool.starmap(resize_img,zip(images,itertools.repeat(img_fldr_dst)))
    # close the pool and wait for the work to finish 
    pool.close() 
    pool.join() 


def resize_img(img_path_src,img_fldr_dest):
    #print("about to resize image=",img_path_src)
    image = io.imread(img_path_src)         
    image = transform.resize(image, [300,300])
    io.imsave(os.path.join(img_fldr_dest,os.path.basename(img_path_src)),image)      
    label = img_path_src[:-4] + '.xml'
    if copyLabels is True and os.path.exists(label) is True :
        copyfile(label,os.path.join(img_fldr_dest,os.path.basename(label)))

setting the argument max_num_of_thread to any number in [1...10] doesn't improve my run time at all ( for 60 images it stays around 30 sec ) , the max_num_of_thread =10 my PC got stuck

my question is : what is the bottle neck in my code , why can't I see any improvement?

some data about my PC :

python -V
Python 3.6.4 :: Anaconda, Inc.


cat /proc/cpuinfo | grep 'processor' | wc -l
4

cat /proc/meminfo 
MemTotal:        8075960 kB
MemFree:         3943796 kB
MemAvailable:    4560308 kB

cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.10

Answer 1

Blame the GIL.

Python has this mechanism called the GIL, global interpreter lock. It is basically a mutex that prevents native threads from executing Python bytecodes at once. This must be done since Python's (at least, CPython) memory management is not thread-safe.

In other words, the GIL will prevent you from running multiple threads at the same time. Essentially, you're running one thread at a time. Multi-threading, in the sense of exploiting multiple CPU cores, is more like an illusion in Python.

Fortunately, there is a way to solve this problem. it's a bit more expensive resource-wise though. You can utilize multiprocessing instead. Python has excellent support for this through the multiprocessing module. This way, you will be able to achieve parallelism ^[1] .

You might ask why isn't multiprocessing affected by the GIL limitations. The answer is pretty simple. Each new process of your program has a different instance (I think there's a better word for this) of the Python interpreter. This means that each process has its own GIL. So, the processes are not managed by the GIL, but by the OS itself. This provides you with parallelism ^[2] .

---

References

• [1] https://softwareengineering.stackexchange.com/questions/186889/why-was-python-written-with-the-gil
• [2] https://jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/

Answer 2

The problem come from the Global Interpreter Lock or GIL. GIL only let one thread run at a time so if you want to do parallel computation use Processing.Pool :

import multiprocessing

pool = multiprocessing.Pool(max_num_of_process)  # Use number of core as max number

!!! multiprocessing.dummy Is a wrapper arround the threading module, it let you interact with threading Pool as you where using Processing Pool.

Answer 3

You should only use multiprocessing with the number of cpu cores you have available. You are also not using a Queue, so the pool of resources are doing the same work. You need to add a queue to your code.

Filling a queue and managing multiprocessing in python