多处理简单功能不起作用，但原因何在

Question

我正在尝试多处理系统命令，但无法让它与一个简单的程序一起工作。 函数runit（cmd）工作正常但...

#!/usr/bin/python3
from subprocess import call, run, PIPE,Popen
from multiprocessing import Pool
import os
pool = Pool()

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

#print(runit('ls -l'))

it = []
for i in range(1,3):
    it.append('ls -l')

results = pool.map(runit, it)

它输出：

Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'runit' on <module '__main__' from './syscall.py'>
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'runit' on <module '__main__' from './syscall.py'>

然后它以某种方式等待并且什么都不做，当我按下Ctrl + C几次时它会吐出：

^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-6:
Traceback (most recent call last):
  File "./syscall.py", line 17, in <module>
Process ForkPoolWorker-5:
    results = pool.map(runit, it)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
...
    buf = self._recv(4)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

Answer 1

我不确定，因为我知道的问题是与Windows相关的（并且我没有访问Linux盒来重新写入），但为了便于携带，你必须在if __name__=="__main__"包装依赖于多处理的命令。 if __name__=="__main__"或它与python生成进程的方式冲突：固定示例在Windows上运行正常（并且在其他平台上也可以正常工作）：

from multiprocessing import Pool
import os

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

#print(runit('ls -l'))

it = []
for i in range(1,3):
    it.append('ls -l')

if __name__=="__main__":
    # all calls to multiprocessing module are "protected" by this directive
    pool = Pool()

（更仔细地研究错误消息，现在我很确定在runit声明之后移动pool = Pool()会在Linux上修复它，但包装在__main__修复+使其可移植）

也就是说，请注意您的多处理只是创建一个新进程，因此您最好使用线程池（ 类似于多处理池的线程池？ ）：创建进程的线程，如下所示：

from multiprocessing.pool import ThreadPool  # uses threads, not processes
import os

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

it = []
for i in range(1,3):
    it.append('ls -l')

if __name__=="__main__":
    pool = ThreadPool()   # ThreadPool instead of Pool
    results = pool.map(runit, it)
    print(results)
        results = pool.map(runit, it)
        print(results)

后一种解决方案更轻巧，并且不易出错（多处理是一个需要处理的精密模块）。 您将能够使用对象，共享数据等...而不需要Manager对象，以及其他优点