Python Multiprocessing Not Working on Windows 10

Question

I am currently working on an image processing project, where I have about 50k images and I want to do several simple preprocesses (change size, change image type, etc.) and save the results into a separate local directory

I have been done the preprocessing code using Python, and it is fine. But it is taking too long and I decided to use the multiprocessing module. But I have encountered a strange behavior when running the code.

Please see below

def doPreprocess(f):
    #do the preprocessing steps and save to directory



if __name__ == '__main__':

    path = "C:/My/Image/Folder/"

    files =  fnmatch.filter(os.listdir(path), pat = '*.jpg')

    print (len(files))

    pool = mp.Pool(processes=4)
    pool.map_async(doPreprocess, files)

    print ("temp")

However, when I run the code with Command Prompt, it skips the preprocessing step and prints temp right away, as if the function didn't run at all. The preprocessed images are not saved either.

Can someone help me?

Thank you in advance.

Note: I am using Windows 10 and Python 3.6

Answer 1

I tested your code and the thing is that your main thread ends before finishing the other threads... if you try this simplifiied version of your code you will see what I mean:

import multiprocessing as mp


def doPreprocess(f):
    print(f)


if __name__ == '__main__':

    files = ["hello"] * 10
    pool = mp.Pool(processes=4)
    res = pool.map_async(doPreprocess, files)

    print ("temp")

This basically prints temp and exit... but if you add:

res.wait()

just before the print("temp") the main thread will wait until all the other ones are finished and then it prints temp and exit. It will also work if you change:

res = pool.map_async(doPreprocess, files)

to:

res = pool.map(doPreprocess, files)

Why? one is non-blocking (map_async) and ideally you do other things in the main thread, and the other ones blocking (map) which is more or less the equivalent of putting the wait() immediately after pool.map_async .