python多处理池，让一个工人执行不同的功能

Question

I have to perform some processing on each line of a file and I have many files present in an input directory. I have to dump the response I get from processing each line (from multiple input files) in to a single result file.

I have decided this flow - Will dump all the input files into a queue and fork 3-4 workers, where each worker works on a unique file, read its content and after processing dump the response into a writer queue. Their will be a separate process which will read this queue and write result in to an output file.

I have comeup with this code-

def write_to_csv(queue):
    file_path = os.path.join(os.getcwd(), 'test_dir', "writer.csv")
    ofile = open(file_path, "w")
    job_writer = csv.writer(ofile, delimiter='\a')
    while 1:
        line = queue.get()
        if line == 'kill':
            print("Kill Signal received")
            break
        if line:job_writer.writerow([str(line).strip()])
    ofile.close()

def worker_main(file_queue, writer_queue):
    print os.getpid(),"working"
    while not file_queue.empty():
        file_name = file_queue.get(True)
        # somewhere in process_file writer_queue.put(line_resp) is called
        # for every line in file_name
        process_file(file_name, writer_queue) 


if __name__ == "__main__":
    file_queue = multiprocessing.Queue()
    output_queue = multiprocessing.Queue()

    writer_pool = multiprocessing.Pool(1, write_to_csv, (output_queue,))

    cwd = os.getcwd()
    test_dir = 'test_dir'
    file_list = os.listdir(os.path.join(cwd, test_dir))
    for file_name in file_list:
        file_queue.put(file_name)

    reader_pool = multiprocessing.Pool(3, worker_main, (file_queue, output_queue))
    reader_pool.close()
    reader_pool.join()

    output_queue.put("kill")

    print("Finished execution")

The code is working fine. But I wonder if it is possible to do the same thing by a single multiprocessing Pool as opposed to using reader_pool and writer_pool in the code above

Answer 1

You could do that by apply_async , also don't set initializer ( write_to_csv or worker_main in your case) when creating Pool object, or it would run the task by default.

file_queue = multiprocessing.Queue()
output_queue = multiprocessing.Queue()

cwd = os.getcwd()
test_dir = 'test_dir'
file_list = os.listdir(os.path.join(cwd, test_dir))
for file_name in file_list:
    file_queue.put(file_name)

pool = Pool(4)

pool.apply_async(write_to_csv, (output_queue,))
[pool.apply_async(worker_main, (file_queue, output_queue, )) for i in range(3)]

pool.close()
pool.join()