我应该使用多处理、多线程还是其他东西？

Question

I am a beginner coder in Python and I'm currently working on a project that I've split into multiple code "modules" for lack of a better term. My code effectively connects to an API, receives data, performs calculations on that data, and then returns communication with the API depending on the calculated values.

Here are the "modules" I've split my code up into:

1. API Authenticator --> is a loop that checks whether or not the current API access code has expired. If it is not the expiration time, nothing happens; if it is the expiration time, the program submits a new access code request and stores the new access code in the computer's memory

2. Data stream --> is an open websockets connection that receives a continuous stream of data from the API

3. Data processor --> takes data from data stream and computes processed values and stores them in the memory.

4. Executor --> if conditions are met using values derived in the data processor, use API key generated from authenticator module and submit request to server to do some action.

I'm effectively interested in running all these modules at once. The trouble is that they are continuous operations that need to be looped over and over again forever, while passing variables in-between them. Is there an effective way to have all of my "modules" run at once and pass data between eachother? --> im looking into multiprocessing and multithreading, but am interested to see if there's a best bet going forward.

Thanks so much.

Answer 1

If you have both network latency and extensive CPU-intensive calculations, you should consider creating both a thread pool and a processing pool. The processing pool is passed to the worker function, which is invoked using the thread pool, and is used to perform the CPU-intensive work. Granted that this is a bit more complicated than just using multiprocessing, but it allows you to overlap the network retrieval more cheaply. The idea is that if you had, for example, 200 URLs to retrieve and 8 processors on your computer, you could create a process pool of size 200 to overlap the retrieval of the 200 URLs as efficiently as possible but, of course, you would only be able to at most process in parallel 8 of the CPU-intensive portions of the submitted tasks. However, creating 200 processes is a relatively expensive operation, especially on platforms that use spawn to create new processes such as Windows (you neglected to tag your question with your platform). It would therefore be more optimal (ie cheaper) to create a thread pool of size 200.

The following is a benchmark simulating the above situation using just a multiprocessing pool versus a thread pool and multiprocessing pool. Unfortunately, on Windows, my platform, you cannot create multiprocessing pools of sizes greater than 60. This means that if you are running under Windows and you had more than 60 URLs to retrieve, you really have no choice but to use the the two types of pools:

from multiprocessing.pool import ThreadPool, Pool
from functools import partial
import time

def cpu_intensive_calculation(data):
    """ takes approximately .25 seconds on my desktop """
    QUARTER_SECOND_ITERATIONS = 5_000_000
    sum = 0
    for _ in range(QUARTER_SECOND_ITERATIONS):
        sum += 1
    return sum

def fetch_data(url):
    """ approximately .3 seconds of "I/O" """
    time.sleep(.3)
    return ""

def retrieve_url1(url):
    retrieved_data = fetch_data(url)
    result = cpu_intensive_calculation(retrieved_data)
    return result

def benchmark1():
    urls = ['x'] * 60
    t1 = time.time()
    pool = Pool(60)
    result = pool.map(retrieve_url1, urls)
    t2 = time.time()
    print('Multiprocessing only:', t2 - t1)

def retrieve_url2(process_pool, url):
    retrieved_data = fetch_data(url)
    result = process_pool.apply(cpu_intensive_calculation, args=(retrieved_data,))
    return result

def benchmark2():
    urls = ['x'] * 60 # 60
    t1 = time.time()
    thread_pool = ThreadPool(60)
    process_pool = Pool()
    result = thread_pool.map(partial(retrieve_url2, process_pool), urls)
    t2 = time.time()
    print('Thread pool and multiprocessing pool:', t2 - t1)

if __name__ == '__main__':
    benchmark1()
    benchmark2()

Prints:

Multiprocessing only: 5.233615875244141
Thread pool and multiprocessing pool: 4.272637844085693

Or use coroutines (package aiohttp from the PyPI repository) for the network retrieval and loop.run_in_executor to execute the CPU-intensive work.

But if the calculations you perform are relatively trivial or are done in Python modules such as numpy that release the Global Interpreter Lock, just use multithreading for everything.

Update

The following includes a benchmark that uses asyncio coroutines instead of multithreading , which can handles hundreds if not thousands of concurrent web retrievals. It uses the ProcessPoolExecutor class from the concurrent.futures module for running "blocking" operations, so I have switched to that class for multiprocessing:

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from functools import partial
import asyncio
import time

def cpu_intensive_calculation(data):
    """ takes approximately .25 seconds on my desktop """
    QUARTER_SECOND_ITERATIONS = 5_000_000
    sum = 0
    for _ in range(QUARTER_SECOND_ITERATIONS):
        sum += 1
    return sum

def fetch_data(url):
    """ approximately .3 seconds of "I/O" """
    time.sleep(.3)
    return ""

def retrieve_url1(url):
    retrieved_data = fetch_data(url)
    result = cpu_intensive_calculation(retrieved_data)
    return result

def benchmark1():
    urls = ['x'] * 60
    t1 = time.time()
    with ProcessPoolExecutor(max_workers=60) as pool:
        result = list(pool.map(retrieve_url1, urls))
    t2 = time.time()
    print('Multiprocessing only:', t2 - t1)

def retrieve_url2(process_pool, url):
    retrieved_data = fetch_data(url)
    future = process_pool.submit(cpu_intensive_calculation, retrieved_data)
    return future.result()

def benchmark2():
    urls = ['x'] * 60 # 60
    t1 = time.time()
    with ThreadPoolExecutor(max_workers=60) as thread_pool:
        with ProcessPoolExecutor() as process_pool:
            result = list(thread_pool.map(partial(retrieve_url2, process_pool), urls))
    t2 = time.time()
    print('Thread pool and multiprocessing pool:', t2 - t1)
    return result

async def retrieve_url3(url):
    await asyncio.sleep(.3) # the I/O portion
    loop = asyncio.get_running_loop()
    result = await loop.run_in_executor(executor, cpu_intensive_calculation, '') # cpu portion
    return result


async def benchmark3():
    global executor
    urls = ['x'] * 60 # 60
    t1 = time.time()
    with ProcessPoolExecutor() as executor:
        result = await asyncio.gather(*(retrieve_url3(url) for url in urls))
    t2 = time.time()
    print('asyncio:', t2 - t1)

if __name__ == '__main__':
    benchmark1()
    benchmark2()
    asyncio.run(benchmark3())

Prints:

Multiprocessing only: 5.816989183425903
Thread pool and multiprocessing pool: 4.408833742141724
asyncio: 4.401129961013794