Python：多处理代码非常慢

Question

I am pulling .8 million of records in one go(this is one time process) from mongodb using pymongo and performing some operation over it .

My code look as below.

    proc = []
    for rec in cursor: # cursor has .8 million rows 
            print cnt
            cnt = cnt + 1
            url =  rec['urlk']
            mkptid = rec['mkptid']
            cii = rec['cii']

            #self.process_single_layer(url, mkptid, cii)


            proc = Process(target=self.process_single_layer, args=(url, mkptid, cii))
            procs.append(proc)
            proc.start()

             # complete the processes
    for proc in procs:
        proc.join()

process_single_layer is a function which is basically downloading urls .from cloud and storing locally.

Now the problem is downloading process is slow as it has to hit a url. And since records are huge to process 1k rows it is taking 6 minutes.

To reduce the time I wanted to implement Multiprocessing . But It is hard to see any difference with above code.

Please suggest me how can I improve the performance in this scenario.

Answer 1

First of all you need to count all the rows in your file and then spawn a fixed number of processes (ideally matching the number of your processor cores), to which you feed via queues (one for each process) a number of rows equal to the division total_number_of_rows / number_of_cores . The idea behind this approach is that you split the processing of those rows between multiple processes, hence achieving parallelism.

A way to find out the number of cores dynamically is by doing:

import multiprocessing as mp
cores_count = mp.cpu_count()

A slight improvement that can be done by avoiding the initial rows count is by adding a row cyclically by creating the list of queues and then apply a cycle iterator on it.

A full example:

import queue
import multiprocessing as mp
import itertools as itools

cores_count = mp.cpu_count()


def dosomething(q):

    while True:

        try:
            row = q.get(timeout=5)
        except queue.Empty:
            break

    # ..do some processing here with the row

    pass

if __name__ == '__main__':
    processes
    queues = []

    # spawn the processes
    for i in range(cores_count):
        q = mp.Queue()
        queues.append(q)
        proc = Process(target=dosomething, args=(q,))
        processes.append(proc)

    queues_cycle = itools.cycle(queues)
    for row in cursor:
        q = next(queues_cycle)
        q.put(row)

    # do the join after spawning all the processes
    for p in processes:
        p.join()

Answer 2

It's easier to use a pool in this scenario.

Queues are not necessary as you don't need to communicate between your spawned processes. We can use the Pool.map to distribute the workload.

Pool.imap or Pool.imap_unordered might be faster with a larger chunk size. (Ref: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.imap ) You can use the Pool.starmap if you want and get rid of tuple unpacking.

from multiprocessing import Pool

def process_single_layer(data):
    # unpack the tuple and do the processing
    url, mkptid, cii = data
    return "downloaded" + url

def get_urls():
    # replace this code: iterate over cursor and yield necessary data as a tuple
    for rec in range(8): 
            url =  "url:" + str(rec)
            mkptid = "mkptid:" + str(rec)
            cii = "cii:" + str(rec)
            yield (url, mkptid, cii)

#  you can come up with suitable process count based on the number of CPUs.
with Pool(processes=4) as pool:
    print(pool.map(process_single_layer, get_urls()))