Python urllib3和代理

Question

I am trying to figure out how to use proxy and multithreading.

This code works:

requester = urllib3.PoolManager(maxsize = 10, headers = self.headers)
thread_pool = workerpool.WorkerPool()

thread_pool.map(grab_wrapper, [item['link'] for item in products])

thread_pool.shutdown()
thread_pool.wait()  

Then in grab_wrapper

requested_page = requester.request('GET', url, assert_same_host = False, headers = self.headers)

Headers consist of: Accept, Accept-Charset, Accept-Encoding, Accept-Language and User-Agent

But this does not work in production, since it has to pass proxy, no authorization is required.

I tried different things (passing proxies to request, in headers, etc.). The only thing that works is this:

requester = urllib3.proxy_from_url(self._PROXY_URL, maxsize = 7, headers = self.headers)
thread_pool = workerpool.WorkerPool(size = 10)

thread_pool.map(grab_wrapper, [item['link'] for item in products])

thread_pool.shutdown()
thread_pool.wait()  

Now, when I run the program, it will make 10 requests (10 threads) and then... stop. No error, no warning whatsoever. This is the only way I can bypass proxy, but it seems like its not possible to use proxy_from_url and WorkerPool together.

Any ideas how to combine those two into a working code? I would rather avoid rewriting it into scrappy, etc. due to time limitation

Regards

Answer 1

first of all I would suggest to avoid urllib like the plague and instead use requests, which has really easy support for proxies: http://docs.python-requests.org/en/latest/user/advanced/#proxies
Next to that, I haven't use it with multi-threading but with multi-processing and that worked really well, the only thing that you have to figure out is whether you have a dynamic queue or a fairly fixed list that you can spread over workers, an example of the latter which spreads a list of urls evenly over x processes:

# *** prepare multi processing
nr_processes = 4
chunksize = int(math.ceil(total_nr_urls / float(nr_processes)))
procs = []
# *** start up processes
for i in range(nr_processes):
    start_row = chunksize * i
    end_row = min(chunksize * (i + 1), total_nr_store)
    p = multiprocessing.Process(
            target=url_loop,
            args=(start_row, end_row, str(i), job_id_input))
    procs.append(p)
    p.start()
# *** Wait for all worker processes to finish
for p in procs:
    p.join()

every url_loop process writes away its own sets of data to tables in a database, so I don't have to worry about joining it together in python.

Edit: On sharing data between processes -> For details see: http://docs.python.org/2/library/multiprocessing.html?highlight=multiprocessing#multiprocessing

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print num.value
    print arr[:]

But as you see, basically these special types (Value & Array) enable sharing of data between the processes. If you instead look for a queue to do a roundrobin like process, you can use JoinableQueue. Hope this helps!

Answer 2

It seems you are discarding the result of the call to thread_pool.map() Try assigning it to a variable:

requester = urllib3.proxy_from_url(PROXY, maxsize=7)
thread_pool = workerpool.WorkerPool(size=10)


def grab_wrapper(url):
    return requester.request('GET', url)


results = thread_pool.map(grab_wrapper, LINKS)

thread_pool.shutdown()
thread_pool.wait()

Note: If you are using a python 3.2 or greater you can use concurrent.futures.ThreadPoolExecutor . It is quote similar to workerpool but is included in the standard library.