与一群工人一起写文件

Question

I have to write some files that are quite big, I'd like to do this in a separate thread (use processor and HDD at the same time) and I'd like to use a pool of workers of size 4 (because there are other operations besides writing, I guesstimate 4 is enough to use the HDD properly without clogging it).

This is the code I have (it does nothing):

import os, os.path, multiprocessing

def asyncFileMake(e):
    with open('files/file%d.txt'%e, 'w') as f:
        print(os.path.dirname(os.path.abspath(os.getcwd())))
        f.write("hi")

def writefile(e, pool):
    pool.apply_async(asyncFileMake, [e])

if __name__ == "__main__":
    if not os.path.exists('files'):
        os.makedirs('files')
    with multiprocessing.Pool(processes=4) as pool:
        for e in range(10):
          writefile(e, pool)

Answer 1

Here's a version which is inspired by your original code snippet. I think the most significant change is the way the function is passed to the pool ( map instead of apply_async , more on that later):

import os
import multiprocessing

def create_file(e):
    with open('files/file%d.txt'%e, 'w') as f:
        print f.name
        f.write("hi")

if __name__ == '__main__':
    if not os.path.exists('files'):
            os.makedirs('files')
    pool = multiprocessing.Pool(processes=4)
    pool.map(create_file, range(10))

The problem with your original implementation is, that pool.apply_async returns a AsyncResult on which you need to call get so that the actual execution gets triggered. I suggest you take another close look at the documentation of multiprocessing . Especially this part about pool workers . Hope that helps.