使用sys.stdout.write和pool.map进行多处理 - 共享sys.stdout

Question

This is probably something very simple I'm missing.

Why can't I use pool.map(sys.stdout.write, iterable) ?

I can use pool.map(len, iterable) using the same iterable but when using sys.stdout.write I get the following exception:

TypeError: expected string or Unicode object, NoneType found

This is the trace:

Traceback (most recent call last):
  File "/home/reut/python/print_mult.py", line 19, in <module>
    pool.map(sys.stdout.write, messages)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
TypeError: expected string or Unicode object, NoneType found

Full code:

#!/usr/bin/env python

import multiprocessing
import sys

# pool of 10 workers
pool = multiprocessing.Pool(10)
messages = ["message #%d\n" % i for i in range(100)]
print messages
pool.map(sys.stdout.write, messages) # doesn't work - error
# print pool.map(len, messages) # works

Edit #1 - ThreadPool works:

When I use ThreadPool (from multiprocessing.pool ) it works, so I suppose it has something to do with not being able to share the sys.stdout stream across processes.

Edit #2 - manual processes works as well:

from multiprocessing import Process
import sys

# pool of 10 workers
processes = []
for i in range(10):
    processes.append(Process(target=sys.stdout.write, args=("I am process %d" % i, )))

for p in processes:
    p.start()

for p in processes:
    p.join()

So now I'm confused because the difference I know between a regular process and a map process is the point it forks. I'm not sure how it's relevant here. The only thing I can think of is that map stores the target internally and is unable to share it with the workers the way the manual constructor of Process does.

Answer 1

The real error is hidden. You can only pass a function that is directly referable from a module namespace. However, in some circumstances there are ways to get around this limitation. Unix has a special feature whereby a process can be forked and all its memory duplicated. This is how instance methods can be 'passed' to a child process -- nothing is actually passed. On the Windows platform processes cannot be forked, but must be spawned instead. This means a new interpreter is started. For the interpreter to run the given function it is sent the name of the function to run and the module it is located in. The interpreter imports the module and looks up the function, before finally running the function.

For a process that is part of a pool, the process has already been started and so it cannot benefit from forking to receive a copy of the appropriate function/method to run. Instead it must use the same technique as when a new process is spawned. This is why you can get your second edit to work, but not the pool to work.

The easiest way to get around your problem is to make print a function rather than a statement.

from __future__ import print_function

import multiprocessing
import sys

if __name__ == '__main__':
    pool = multiprocessing.Pool(2)
    messages = ["message #%d\n" % i for i in range(5)]
    print(messages) # <- notice the brackets around the arguments to print
    pool.map(print, messages)

Failing that you can define a function that will do the printing for you, and use that as the function for map.

import multiprocessing 
import sys

def stdout_write(arg):
    sys.stdout.write(arg)

def stdout_print(arg):
    print arg

if __name__ == '__main__':
    pool = multiprocessing.Pool(2)
    messages = ["message #%d\n" % i for i in range(5)]
    print messages
    pool.map(stdout_print, messages)

Answer 2

I'm not sure why, exactly, but pool.map() is requiring the function to return a string.

This simple change to your program runs correctly.

#!/usr/bin/env python

import multiprocessing
import sys

def prn(s):
    sys.stdout.write(s)
    return ''

# pool of 10 workers
pool = multiprocessing.Pool(10)
messages = ["message #%d\n" % i for i in range(100)]
print messages
pool.map(prn, messages) # doesn't work - error
# print pool.map(len, messages) # works

I checked the documentation and I don't see this requirement so I don't know why it is being enforced.