共享输出的python多重处理

Question

I am trying to process a large number of text files and calculating data within them (simple addition). The problem is this takes a long time, and know that there are some multiprocessing functions in other languages, but have never done anything like this in Python.

Let's say I have a directory with 16,000 files. Currently, I open each file individually, bring it into an array in Python, do some manipulation of the data, and then output to a master array (with length 16,000). Can a multiprocessing function be utilized to run several instances of 'opening the file, processing the data, and output information' to the same array?

The original code is basically like this:

# path
filepath = /path/to/file

# Get the dir contents
filedir = os.listdir(filepath)

# Pre-allocate large array
large_array = np.zeros(len(filedir))

# Begin loop
for i in range(0,len(filedir)):
    # Define the path to load the text file
    filename = filepath + '/' + filedir[i]

    output = []
    output = function_to_process_filename(filename)

    large_array[i] = output

Where would the multiprocessing / parallel portion go to potentially make the code run faster and what does it look like in Python?

Answer 1

You can use a multiprocessing Pool to submit work to a pool of processes.

The map function will take an iterable and split it into chunks of work, on which your function can be applied (see here ):

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

For your example, you could pass a list of file names the map function and a function that will open the file and manipulate it. You can pass the processed file contents as result and concatenate everything in the main process.

Answer 2

So if I understand correctly, what you are looking for is a way to process several jobs in parallel with multiprocessing and each job filling a single python data structure containing the results ? I will complete the good previous answer by using indeed map but also multiprocessing.Manager() :

from multiprocessing import Pool, Manager, ctypes, cpu_count
from functools import partial

# path to dir
dir_path = /path/to/dir

# Get the dir content
files = os.listdir(dir_path)

def processing_func(results_array, filename):
    # process filename
    # add element to results_array

NB_CPU = cpu_count()
# change ctype with what the array will contain
results_array = Manager().array(ctypes.c_int, len(files))
with Pool(processes=NB_CPU) as pool:
    # this is used to pass args to multiprocessed function
    function_with_args = partial(processing_func, results_array)
        # this will iterate through the files and fill NB_CPU processes at a time
        # by applying the function_with_args on each iterated element
        pool.map(function_with_args, files)