Python 多处理列表问题

Question

I am attempting to dynamically open and parse through several text files (~10) to extract a particular value from key, for which I am utilizing multi-processing within Python to do this. My issue is that the function that I am calling writes particular data to a class list which I can see in the method, however outside the method that list is empty. Refer to the following:

class:

class MyClass(object):
    __id_list = []

    def __init__(self):
        self.process_wrapper()

Caller Method:

def process_wrapper(self):
    from multiprocessing import Pool
    import multiprocessing

    info_file = 'info*'
    file_list = []

    p = Pool(processes = multiprocessing.cpu_count() - 1)

    for file_name in Path('c:/').glob('**/*/' + info_file):
        file_list.append(str(os.path.join('c:/', file_name)))

    p.map_async(self.get_ids, file_list)

    p.close()
    p.join()

    print(self.__id_list) # this is showing as empty

Worker method:

def get_ids(self, file_name):        
    try:
        with open(file_name) as data:
            for line in data:
                temp_split = line.split()
                for item in temp_split:
                    value_split = str(item).split('=')
                    if 'id' == value_split[0].lower():
                        if int(value_split[1]) not in self._id_list:
                            self.__id_list.append(int(value_split[1]))
    except:
        raise FileReadError(f'There was an issue parsing "{file_name}".')
    print(self.__id_list) # here the list prints fine

Answer 1

The map call returns a AysncResult class object. you should use that to wait for the processing to finish before checking self.__id_list. also you might consider returning a local list, collected those lists and aggregating them into the final list.

Answer 2

1. It looks like you have a typo in your get_ids method ( self._id_list instead of self.__id_list ). You can see it if you wait for the result:

result = p.map_async(self.get_ids, file_list)
result.get()

2. When a new child process is created, it gets a copy of the parent's address space however any subsequent changes (either by parent or child) are not reflected in the memory of the other process. They each have their own private address space.

Example:

$ cat fork.py 
import os

l = []
l.append('global')

# Return 0 in the child and the child’s process id in the parent
pid = os.fork()

if pid == 0:
    l.append('child')
    print(f'Child PID: {os.getpid()}, {l}')
else:
    l.append('parent')
    print(f'Parent PID: {os.getpid()}, {l}')

print(l)

$ python3 fork.py 
Parent PID: 9933, ['global', 'parent']
['global', 'parent']
Child PID: 9934, ['global', 'child']
['global', 'child']

Now back to your problem, you can use multiprocessing.Manager.list to create an object that is shared between processes:

from multiprocessing import Manager, Pool

m = Manager()
self.__id_list = m.list()

Docs: Sharing state between processes

or use threads as your workload seems to be I/O bound anyway:

from multiprocessing.dummy import Pool as ThreadPool

p = ThreadPool(processes = multiprocessing.cpu_count() - 1)

Alternatively check concurrent.futures