在Python多处理中处理多个结果

Question

I'm writing a Python piece of code to parse a lot of ascii file using multiprocessing functionality. For each file I've to perform the operations of this function

def parse_file(file_name):
    record = False
    path_include = []
    buffer_include = []
    include_file_filters = {}
    include_keylines = {}
    grids_lines = []
    mat_name_lines = []
    pids_name_lines = []
    pids_shell_lines= []
    pids_weld_lines = []
    shells_lines = []
    welds_lines = []
    with open(file_name, 'rb') as in_file:
        for lineID, line in enumerate(in_file):
            if record:
                path_include += line
            if record and re.search(r'[\'|\"]$', line.strip()):
                buffer_include.append(re_path_include.search(
                    path_include).group(1).replace('\n', ''))
                record = False
            if 'INCLUDE' in line and '$' not in line:
                if re_path_include.search(line):
                    buffer_include.append(
                        re_path_include.search(line).group(1))
                else:
                    path_include = line
                    record = True
            if line.startswith('GRID'):
                grids_lines += [lineID]
            if line.startswith('$HMNAME MAT'):
                mat_name_lines += [lineID]
            if line.startswith('$HMNAME PROP'):
                pids_name_lines += [lineID]
            if line.startswith('PSHELL'):
                pids_shell_lines += [lineID]
            if line.startswith('PWELD'):
                pids_weld_lines += [lineID]
            if line.startswith(('CTRIA3', 'CQUAD4')):
                shells_lines += [lineID]
            if line.startswith('CWELD'):
                welds_lines += [lineID]
    include_keylines = {'grid': grids_lines, 'mat_name': mat_name_lines, 'pid_name': pids_name_lines, \
                        'pid_shell': pids_shell_lines, 'pid_weld': pids_weld_lines, 'shell': shells_lines, 'weld': welds_lines}
    include_file_filters = {file_name: include_keylines}
    return buffer_include, include_file_filters 

This function is used in a loop through list of files, in this way (each process on CPU parse one entire file)

import multiprocessing as mp
p = mp.Pool(mp.cpu_count())
buffer_include = []
include_file_filters = {}
for include in grouper([list_of_file_path]):
    current = mp.current_process()
    print 'Running: ', current.name, current._identity
    results = p.map(parse_file, include) 
    buffer_include += results[0]
    include_file_filters.update(results[1])
p.close()

The grouper function used above is defined as

def grouper(iterable, padvalue=None):
    return itertools.izip_longest(*[iter(iterable)]*mp.cpu_count(), fillvalue=padvalue)

I'm using Python 2.7.15 in cpu with 4 cores (Intel Core i3-6006U).

When I run my code, I see all the CPUs engaged on 100%, the output in Python console as Running: MainProcess () but nothing appened otherwise. It seems that my code is blocked at instruction results = p.map(parse_file, include) and can't go ahead (the code works well when i parse the files one at a time without parallelization).

• What is wrong?
• How can I deal with the results given by parse_file function during parallel execution?My approach is correct or not?

Thanks in advance for your support

EDIT

Thanks darc for your reply. I've tried your suggestion but the issue is the same. The problem, seems to be overcome if I put the code under if statement like so

if __name__ == '__main__':

Maybe this is due to the manner in which Python IDLE handle the process. I'm using the IDLE environ for development and debugging reasons.

Answer 1

according to python docs :

map(func, iterable[, chunksize]) A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

since it is blocking your process wait until parse file is done.

since map already chnucks the iterable you can try to send all of the includes together as one large iterable.

import multiprocessing as mp
p = mp.Pool(mp.cpu_count())
buffer_include = []
include_file_filters = {}
results = p.map(parse_file, list_of_file_path, 1) 
buffer_include += results[0]
include_file_filters.update(results[1])
p.close()

if you want to keep the original loop use apply_async, or if you are using python3 you can use ProcessPoolExecutor submit() function and read the results.