如何在Python中加快网络抓取？

Question

I'm using urllib.urlopen() method and BeautfulSoup for crawling. I am not satisfied with browsing speed, and I'm thinking about what urllib is parsing, guessing it got to load more than html only. Couldn't find in docs does it read or checks bigger data (images, flash, ...) by default.

So, If urllib has to load ie images, flash, js... how to avoid GET request for such data types?

Answer 1

Try requests - it implements HTTP connection pools which speeds up the crawling.

Also, it takes care of other things like cookies, auth etc. much better than urllib and works great with BeautfulSoup..

Answer 2

Use threading! It's super simple. Here's an example. You can change the number of connections to suit your needs.

import threading, Queue
import urllib

urls = [
    'http://www.google.com',
    'http://www.amazon.com',
    'http://www.ebay.com',
    'http://www.google.com',
    'http://www.amazon.com',
    'http://www.ebay.com',
    'http://www.google.com',
    'http://www.amazon.com',
    'http://www.ebay.com',    
    ]

queue = Queue.Queue()
for x,url in enumerate(urls):
    filename = "datafile%s-%s" % (x,url)
    queue.put((url, filename))


num_connections = 10

class WorkerThread(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self):
        while 1:
            try:
                url, filename = self.queue.get_nowait()
            except Queue.Empty:
                raise SystemExit

            urllib.urlretrieve(url,filename.replace('http://',''))

# start threads
threads = []
for dummy in range(num_connections):
    t = WorkerThread(queue)
    t.start()
    threads.append(t)


# Wait for all threads to finish
for thread in threads:
    thread.join()