How to print an error message which appeared while multiprocessing (not raise just print ) on using Pool().map()

Question

I am downloading images using multiprocessing and a unique thing I noticed that you can not get Error Message except there is BaseException . For simple cases, you can run a loop and do the following:

for i in range(start, start+end):
        url = df.iloc[i,ind]
        try:
            
            load_save_image_from_url(url,DIR,str(i),resize=resize,resize_shape=resize_shape)
            count+=1

        except (KeyboardInterrupt, SystemExit):
            sys.exit("Forced exit prompted by User: Quitting....")

        except Exception as e:
            print(f"Error at index {i}: {e}\n")
            pass

It works completely fine But when you use Multiprocessing you can not either use logging or print because just a log or print is described which is last one.

try:
        pool = Pool(workers)
        pool.map(partial(load_save_image_from_url,OUT_DIR=DIR,resize=resize,resize_shape=resize_shape),
        lis_tups)

    except (KeyboardInterrupt, SystemExit):
        sys.exit("Forced exit prompted by User: Quitting....")

    except ConnectionError:
        logging.error(f"Connection Error for some URL")
        pass

    except Exception as e:
        logging.error(f'Some Other Error most probably Image related')
        pass
pool.close()
pool.join()

Using pool.get() can work but it has to be in a loop and that too at the end of program.

How can I print an error or log an error when there is an exception while being in multiprocessing?

Answer 1

You can catch the exception inside the load_image function itself and return it to the main process.

result = pool.map(load_image, ...)
if result instanceof Exception:
    # handle it
else:
  # image loading succeded


def load_image():
  try:
      # make a get request
      return image
  except Exception as e:
      return e