我如何在Selenium Web驱动程序python中使用多线程进行Web爬网

Question

i have scenario where i go to a webpage and open each link in a new window and check for specific documents but its taking enormous time to go through each link so is there a way to increase performance using multi-threading

known problem

selenium is not thread safe

but i can create multiple instances of driver for each thread which takes care of this problem

current code which i am using

    for tag in self.driver.find_elements_by_xpath('//body//a[@href]'):
        current_window = self.driver.current_window_handle
        if href:
                self.driver.execute_script('window.open(arguments[0]);', href)
                time.sleep(10)
                new_window = [window for window in self.driver.window_handles if window != current_window][0]
                self.driver.switch_to.window(new_window)
                # Execute required operations
                func_url=self.driver.current_url
                self.driver.close()
                self.driver.switch_to.window(current_window)
                if not func_url:
                    continue
                if re.search('\.(?:pdf|png|jpg|doc|ppt|zip)',func_url):
                        cat=fd.findCat(func_url)
                        fd.findDate(func_url)
                time.sleep(10)

Answer 1

If you want to win time, first of all you have to use WebDriverWait:

This is an EXAMPLE

def exec_sync(driver, script, waitTime, scriptName):
    global contador
    print scriptName
    driver.execute_script(script)
    WebDriverWait(driver, waitTime).until(
        EC.presence_of_element_located(
            (By.ID, 'uniqueIdentifier' + str(contador))
        )
        #driver.find_element_by_id(cadena)
    )
    contador += 1
    print 'Success'

Second, WebDriver is not for multithreading as you have said, so you HAVE TO use SEMAPHORE

https://docs.python.org/2/library/threading.html#semaphore-objects https://www.tutorialspoint.com/python/python_multithreading.htm

It's the only way for multithreading

MT + Wait.until is what you need to save time