Python Multiprocessing within Jupyter Notebook does not work

Question

I am new to multiprocessing module in Python and work with Jupyter notebooks. When I try to run the following code I keep getting AttributeError: Can't get attribute 'load' on <module '__main__' (built-in)>

When I run the file there is no output, it just keeps loading.

import pandas as pd
import datetime
import urllib
import requests
from pprint import pprint
import time
from io import StringIO
from multiprocessing import Process, Pool

symbols = ['AAP']

start = time.time()
dflist = []

def load(date):
    if date is None:
        return
    url = "http://regsho.finra.org/FNYXshvol{}.txt".format(date)
    try:
        df = pd.read_csv(url,delimiter='|')
        if any(df['Symbol'].isin(symbols)):
            stocks = df[df['Symbol'].isin(symbols)]
            print(stocks.to_string(index=False, header=False))
            # Save stocks to mysql
        else:
            print(f'No stock found for {date}' )
    except urllib.error.HTTPError:
        pass

pool = []
numdays = 365
start_date = datetime.datetime(2019, 1, 15 )  #year - month - day
datelist = [
        (start_date - datetime.timedelta(days=x)).strftime('%Y%m%d') for x in range(0, numdays)
        ]

pool = Pool(processes=16)
pool.map(load, datelist)

pool.close()
pool.join()

print(time.time() - start)

What can I do to run this code directly from the notebook without issues?

Answer 1

one way to do it:
1. get load function out and create for example worker.py
2. import worker and worker.load
3.

from multiprocessing import Pool
import worker
if __name__ ==  '__main__': 
  pool = []
  numdays = 365
  start_date = datetime.datetime(2019, 1, 15 )  #year - month - day
  datelist = [
        (start_date - datetime.timedelta(days=x)).strftime('%Y%m%d') for x in 
        range(0, numdays)
        ]

  pool = Pool(processes=16)
  pool.map(worker.load, datelist)

  pool.close()
  pool.join()