在多个进程之间分配python工作负载

Question

Let us suppose that I want to do a google search for the word 'hello'. I then want to go to every single link on the first 100 pages of Google and download the HTML of that linked page. Since there are 10 results per page, this would mean I'd have to click about 1,000 links.

This is how I would do it with a single process:

from selenium import webdriver
driver=webdriver.Firefox()
driver.get('http://google.com')

# do the search
search = driver.find_element_by_name('q')
search.send_keys('hello')
search.submit()

# click all the items
links_on_page = driver.find_elements_by_xpath('//li/div/h3/a')
for item in links_on_page:
    item.click()
    # do something on the page
    driver.back()

# go to the next page
driver.find_element_by_xpath('//*[@id="pnnext"]')

This would obviously take a very long time to do this on 100 pages. How would I distribute the load, such that I could have (for example) three drivers open, and each would 'check out' a page. For example:

• Driver #1 checks out page 1. Starts page 1.
• Driver #2 sees that page 1 is checked out and goes to page #2. Starts page 2.
• Driver #3 sees that page 1 is checked out and goes to page #2. Same with page 2. Starts page 3.
• Driver #1 finishes work on page 1...starts page 4.

I understand the principle of how this would work, but what would be the actual code to get a basic implementation of this working?

Answer 1

You probably want to use a multiprocessing Pool . To do so, write a method that is parameterised by page number:

def get_page_data(page_number):
    # Fetch page data
    ...
    # Parse page data
    ...
    for linked_page in parsed_links:
        # Fetch page source and save to file
        ...

Then just use a Pool of however many processes you think is appropriate (determining this number will probably require some experimentation):

from multiprocessing import Pool

if __name__ == '__main__':
    pool = Pool(processes=4)
    pool.map(get_page_data, range(1,101))

This will now set 4 processes going, each fetching a page from Google and then fetching each of the pages it links to.

Answer 2

Not answering your question directly, but proposing an avenue that might make your code usable in a single process, thereby avoiding you synchronisation issues between different threads/processes...

You would probably better be with a framework such as Twisted which enables asynchronous network operations, in order to keep all operations within the same process. In your code, the parsing of the HTML code is likely going to take far less time than the complete network operations required to fetch the pages. Therefore, using asynchronous IO, you can launch a couple of requests at the same time and parse the result only as responses arrive. In effect, each time a page is returned, your process is likely to be "idling" in the runloop.