Python multiprocess Pool vs Process

Question

I'm new to Python multiprocessing. I don't quite understand the difference between Pool and Process. Can someone suggest which one I should use for my needs?

I have thousands of http GET requests to send. After sending each and getting the response, I want to store to response (a simple int) to a (shared) dict. My final goal is to write all data in the dict to a file.

This is not CPU intensive at all. All my goal is the speed up sending the http GET requests because there are too many. The requests are all isolated and do not depend on each other.

Shall I use Pool or Process in this case?

Thanks!

----The code below is added on 8/28---

I programmed with multiprocessing. The key challenges I'm facing are:

1) GET request can fail sometimes. I have to set 3 retries to minimize the need to rerun my code/all requests. I only want to retry the failed ones. Can I achieve this with async http requests without using Pool?

2) I want to check the response value of every requests, and have exception handling

The code below is simplified from my actual code. It is working fine, but I wonder if it's the most efficient way of doing things. Can anyone give any suggestions? Thanks a lot!

def get_data(endpoint, get_params):
    response = requests.get(endpoint, params = get_params)
    if response.status_code != 200:
        raise Exception("bad response for " + str(get_params))
    return response.json()

def get_currency_data(endpoint, currency, date):
    get_params = {'currency': currency,
                  'date' : date
                  }
    for attempt in range(3):
        try:
            output = get_data(endpoint, get_params)
            # additional return value check
            # ......
            return output['value']
        except:
            time.sleep(1) # I found that sleeping for 1s almost always make the retry successfully
    return 'error'

def get_all_data(currencies, dates):
    # I have many dates, but not too many currencies
    for currency in currencies:
        results = []
        pool = Pool(processes=20)
        for date in dates:
            results.append(pool.apply_async(get_currency_data, args=(endpoint, date)))
        output = [p.get() for p in results]
        pool.close()
        pool.join()
        time.sleep(10) # Unfortunately I have to give the server some time to rest. I found it helps to reduce failures. I didn't write the server. This is not something that I can control

Answer 1

Neither. Use asynchronous programming . Consider the below code pulled directly from that article (credit goes to Paweł Miech)

#!/usr/local/bin/python3.5
import asyncio
from aiohttp import ClientSession

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()

async def run(r):
    url = "http://localhost:8080/{}"
    tasks = []

    # Fetch all responses within one Client session,
    # keep connection alive for all requests.
    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(fetch(url.format(i), session))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)
        # you now have all response bodies in this variable
        print(responses)

def print_responses(result):
    print(result)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(4))
loop.run_until_complete(future)

Just maybe create a URL's array, and instead of the given code, loop against that array and issue each one to fetch .

---

EDIT: Use requests_futures

As per @roganjosh comment below, requests_futures is a super-easy way to accomplish this.

from requests_futures.sessions import FuturesSession
sess = FuturesSession()
urls = ['http://google.com', 'https://stackoverflow.com']
responses = {url: sess.get(url) for url in urls}
contents = {url: future.result().content 
            for url, future in responses.items()
            if future.result().status_code == 200}

---

EDIT: Use grequests to support Python 2.7

You can also us grequests, which supports Python 2.7 for performing asynchronous URL calling.

import grequests
urls = ['http://google.com', 'http://stackoverflow.com']
responses = grequests.map(grequests.get(u) for u in urls)
print([len(r.content) for r in rs])
# [10475, 250785]

---

EDIT: Using multiprocessing

If you want to do this using multiprocessing, you can. Disclaimer: You're going to have a ton of overhead by doing this, and it won't be anywhere near as efficient as async programming... but it is possible.

It's actually pretty straightforward, you're mapping the URL's through the http GET function:

import requests
urls = ['http://google.com', 'http://stackoverflow.com']
from multiprocessing import Pool
pool = Pool(8)
responses = pool.map(requests.get, urls)

The size of the pool will be the number of simultaneously issues GET requests. Sizing it up should increase your network efficiency, but it'll add overhead on the local machine for communication and forking.

Again, I don't recommend this, but it certainly is possible, and if you have enough cores it's probably faster than doing the calls synchronously.