带有pandas的Python多重处理并非所有进程都同时运行

Question

I am reading a csv in chunk and passing the chunk to a pool of 4 processes.

pool = Pool(processes=4)
            chunk_index = 1
            for df in pd.read_csv(downloaded_file, chunksize=chunksize, compression='gzip', skipinitialspace=True, encoding='utf-8'):
                output_file_name = output_path + merchant['output_file_format'].format(
                    file_index, chunk_index)
                pool.map(wrapper_process, [
                         (df, transformer, output_file_name)])
                chunk_index += 1

With this code my understanding is it should show me 4 process continuously running. But in the screenshot of htop below, It is always 2 running. One is htop command it self. It means that only 1 python process in running at the time.

From the memory usage, It is 12 gb which i think will only be possible when the 4 chunks are loaded in memory provided 1 chunk is 2gb almost

How can i use for processors at once.

Answer 1

The problem is that you misuderstood how map works. From the doc :

map(func, iterable[, chunksize]) This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

As iterable you provide a list with only one element: the tuple (df, ...) . But you'd need to provide a iterable with many elements. To make this work, you'd need to prepare the list first and only then send it to the processes (hint: you can just write Pool() and let python find out the number of cores itself)

pool = Pool()
chunk_index = 1
list = []
for df in pd.read_csv(downloaded_file, 
        chunksize=chunksize, 
        compression='gzip', 
        skipinitialspace=True, 
        encoding='utf-8'):
    output_file_name = output_path + 
        merchant['output_file_format'].format(file_index, chunk_index)
    list.append((df, transformer, output_file_name)])
    chunk_index += 1
pool.map(wrapper_process, list)

But now you have the problem that you need to hold the full csv data in memory which might be ok, but is usually not. To come around this problem you could switch to using a queue: You would

• build up an empty queue
• start the processes and tell them to get items from the queue (which is still empty at start)
• feed the queue with your main process (and maybe check that the queue is not getting too long so memory consumption doesn't go into the roof)
• put a STOP element to the queue so the processes quit themselves

There's a good example in the official doc (look at the last example on the page) which explains you would approach that.

One last word: Are you sure your operation is CPU bound? Do you do a lot of processing in wrapper_process (and possibly also transformer )? Because if you just split the CSV in separate files without much processing your program is IO bound and not CPU bound and then the multiprocessing wouldn't make any sense.