使用Selenium和Multiprocessing进行Python Web抓取

Question

I have the following function.It takes a link as an argument, and gives me the data I need at the end (base_list)

def extract_BS(link):

    base_list = []

    try:

        browser = webdriver.Firefox()
        browser.get(link)                   
        print('Processing..' + link)

        respData = browser.page_source
        soup = bs.BeautifulSoup(respData, 'html.parser')
        browser.quit()
        shipping_options = soup.find("div", class_ = "modal_shipping_con").find_all("input", class_="inputChangePrice")
        base_list.append(shipping_options)

    except Exception as e:
        print(urls)
        print('Exception:',e)
        pass

    return base_list

However, my list of urls is around 1000, so it takes too long to do them individually. I found this blog on multiprocessing, which I am trying to tweak

My additional code to use the multithreading is as follows

with Pool(4) as p:
    records = p.map(extract_BS,urls)

p.terminate()
p.join()

However, when I run it, I get the following error.

MaybeEncodingError: Error sending result:
Reason: 'RecursionError('maximum recursion depth exceeded in       comparison',)'

When I run the script, 4 windows pop open, and the webscraper does its thing, but it seems like the problem lies with 'records = p.map(extract_BS,urls)'. My function outputs a list, which I want to store in 'results', but for some reason this is not working.

Any advice/help/observations much appreciated, thanks

Answer 1

You need to place

if __name__ == "__main__":

before with Pool(4) as p to prevent to create a pool in each pool which causes the recursion error.