与硒多线程find_element

Question

I'm trying to lower execution time by multithreading the more time consuming parts of my script, which is mostly the locator calls.

However, I keep getting "CannotSendRequest" and "ResponseNotReady" exceptions from the two threads.

Is this because I'm using the same http handle?

input_worker = threading.Thread(name="input_worker", target=find_input_fields, args=(form, args, logger))
input_worker.setDaemon(True)
select_worker = threading.Thread(name="select_worker", target=find_select_fields, args=(form, logger))
select_worker.setDaemon(True)

thread_pool.append(input_worker)
thread_pool.append(select_worker)

And in the find_input_fields function is something like

input_fields = form.find_elements_by_tag_name("input")

Answer 1

Selenium takes 1 cpu core per thread. And multi threading is not suggested for Selenium webdriver. Consider If you have a 4 core system you can run 4 selenium separate thread linked to each core. As you are creating 2 threads you are getting Exceptions from the two threads.

FYI

Is it possible to parallelize selenium webdriver get_attribute calls in python?