How to use Multiprocessing with Selenium in Python

Question

I am trying to use multiprocessing using selenium in python. My code is as follows:

from selenium import webdriver
from multiprocessing import Pool
import xlwings as xw

driver = webdriver.Chrome('chromedriver.exe')
driver.get("https://example.com")

wb = xw.Book('my_file.xlsm')
sht = wb.sheets["Sheet1"]
final_list = []

search = driver.find_element_by_id("ContentPlaceHolder1_txtByName")
for item in search:
       z = item.find_element_by_class_name("valuetext")
       info = z.find_element_by_tag_name("span")
       final_list.append(info.text)

def automate(num):
    col = num
    list_item = final_list[num]   
    sht.range(1, col).value = each


if __name__ == '__main__':

    p = Pool(processes=4)
    data = p.map(automate,range(1,20))        

The issue I'm having is for each of the 4 processes the web page is re-opened again and I don't understand why. If p.map is only targeting the automate function then why is the rest of the code run for every process?

I'm still new to multiprocessing so am not sure if that's just how it works.Is there another way to do this to ensure the processes only target the function itself, or is there a way I could use threading?

Answer 1

In the examples in the multiprocessing docs , they suggest using Pool with a context manager, ie,

with Pool(processes=4) as pool: print(pool.map(f, range(10))

The's the most stand-out different I immediately see between your use and the docs. I don't see it spelled out, but I would infer based on your observation, that the framework is "reimporting" (so to speak) your module in each process that it spawned, and that's resulting in the behavior you report: namely, multiple browsers opening.

To prevent that, I would recommend putting the initialization code within a function; if you want to share the final_list , you should probably do so with a queue or other data structure supported by multiprocess .