Python Multiprocessing disallows printing in main process after processes are created

Question

I was trying to make sure I had this code setup properly before starting the processes. After adding some print statements I found that only 'outer' and 'inner' are printing and I can not understand why the other print statements are not executing.

import multiprocessing
from itertools import product

retailer_ids = [41, 499]  # defined retailers
product_ids = [4, 5, 10, 11, 12]  # ProductIDs to search on
NUMBER_OF_PROCESSES = 2

retailer_products = list(product(retailer_ids, product_ids))

# Start processing the retailer/product combinations
for i in range(0, len(retailer_products), NUMBER_OF_PROCESSES):
    print('outer')
    try:
        current_processes = []
        for j in range(0, NUMBER_OF_PROCESSES):
            print('inner')
            process = multiprocessing.Process(scrape_retailer_product, retailer_products[i+j])
            #process.start()
            current_processes.append(process)
        # wait for current process to finish before starting more
        print('waiting for processes to complete')
        for p in current_processes:
            p.join()

        print('completed')

    # something bad happened during process creation or a
    # a scrape process returned with an exception it could not handle
    except Exception as e:
        for p in current_processes:
            p.terminate()
            print('term')
            exit()

Answer 1

The problem is your are catching all exceptions. So your code was not passing the correct arguments to the Process constructor (which generated an AssertionError ), but your catch statement was silently handling the exception.

The current exception is:

Traceback (most recent call last):
  File "C:\Users\MiguelAngel\Downloads\test.py", line 19, in <module>
    process = multiprocessing.Process(scrape_retailer_product, args=(retailer_products[i+j]))
  File "C:\Users\MiguelAngel\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\process.py", line 82, in __init__
    assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now

I suppose that scrape_retailer_product is the function that should be executed in the new process. Therefore, according to the documentation , the call to the constructor should be:

process = multiprocessing.Process(target=scrape_retailer_product, 
                                  args=(retailer_products[i+j],))

If you want to catch all multiprocessing exceptions, you should catch multiprocessing.ProcessError . According to the documentation , it is the base class of all multiprocessing exceptions.