Python-并发运行函数（多个实例）

Question

I built a little function that will gather some data using a 3rd party API. Call if def MyFunc(Symbol, Field) that will return some info based on the symbol given.

The idea was to fill a Pandas df with the returned value using something like:

df['MyNewField'] = df.apply(lamba x: MyFunc(x, 'FieldName'))

All this works BUT , each query takes around 100ms to run. This seems fast until you realize you may have 30,000 or more to do (3,000 Symbols with 10 fields each for starters).

I was wondering if there would be a way to run this concurrently as each request is independent? I am not looking for multi processor etc. libraries but instead a way to do multiple queries to the 3rd party at the same time to reduce the time taken to gather all the data. (Also, I suppose this will change the initial structure used to store all the received data - I do not mind not using Apply and my dataframe at first and instead save the data as it is received on a text or library type structure -).

NOTE: While I wish I could change MyFunc to request multiple symbols/fields at once this cannot be done for all cases (meaning some fields do not allow that and a single request is the only way to go). This is why I am looking at concurrent execution and not at changing MyFunc.

Thanks!

Answer 1

There are many libraries to parallelize pandas dataframe. However, I prefer native multi-processing pool to do the same. Also, I use tqdm along with it to know the progress.

import numpy as np
from multiprocessing import cpu_count, Pool

cores = 4 #Number of CPU cores on your system
partitions = cores #Define as many partitions as you want

def partition(data, num_partitions):
    partition_len = int(len(data)/num_partitions)
    partitions = []

    num_rows = 0
    for i in range(num_partitions-1):
        partition = data.iloc[i*partition_len:i*partition_len+partition_len]
        num_rows = num_rows + partition_len
        partitions.append(partition)

    partitions.append(data.iloc[num_rows:len(data)])
    return partitions

def parallelize(data, func):
    data_split = partition(data, partitions)
    pool = Pool(cores)
    data = pd.concat(pool.map(func, data_split))
    pool.close()
    pool.join()
    return data

df['MyNewField'] = parallelize(df['FieldName'], MyFunc)