[英]Python multiprocessing.Pool is weird on Windows
我有一个超简单的python脚本,如这里定义
import multiprocessing
from multiprocessing import Pool
print "CPUs: " + str(multiprocessing.cpu_count())
convertPool = Pool(multiprocessing.cpu_count())
在Linux上,这似乎与我预期的一样,程序只是启动,然后打印出内核数量,然后退出。 但是,在Windows上,该程序将继续调用新的python命令并打印出“ CPU:4”(我有4个内核),并且永不停止运行,最终将杀死该计算机。 有人可以解释一下这是怎么回事吗?
谢谢
编辑:我现在有以下程序仍然无法正常运行
import sys
import os
import subprocess
from subprocess import Popen, PIPE
from threading import Thread
import multiprocessing
from multiprocessing import Pool, freeze_support
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # python 3.x
ON_POSIX = "posix" in sys.builtin_module_names
def myPrint(line, logWriter):
if logWriter is None:
# This will print without a newline, cause the process
# has its own newlines
sys.stdout.write(line)
else:
logWriter.write(line.strip())
logWriter.write("\n")
logWriter.flush()
# This is gotten from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
def executeCommand(cmd, logWriter):
myPrint(cmd + "\n", logWriter)
p = Popen(cmd, stdout=PIPE, bufsize=4096, close_fds=ON_POSIX, shell=True)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
# read line without blocking
while t.isAlive() or not q.empty():
try:
line = q.get_nowait() # or q.get(timeout=.1)
except Empty:
pass # Do nothing
else: # If there is a line, then print it
if logWriter is not None:
myPrint(line, logWriter)
# Sleep to prevent python from using too much cpu
time.sleep(0.05)
if __name__ == "__main__":
freeze_support()
convertPool = Pool(multiprocessing.cpu_count())
# Now let's smooth and threshold all the masks
for foo in bar:
threshSmoothCmd = "ConvertImage.exe -i " + foo + " -o " + foo
myPrint(threshSmoothCmd + "\n", None)
convertPool.apply_async(executeCommand,(threshSmoothCmd, None,))
convertPool.close()
convertPool.join()
print "Finished processing"
ConvertImage.exe是我自己写的可执行文件foo和bar只是占位符。 在Linux上,这将启动multiprocessing.cpu_count()个ConvertImage.exe进程的数量,然后在所有ConvertImage.exe进程完成后打印“ Finished”处理。 在Windows上,这立即启动len(bar)ConvertImage.exe进程,然后立即打印Finished处理并退出。 如何使Windows版本的行为与Linux版本类似?
我发现了这一点,它工作正常,由于忘记了一些导入操作,导致示例程序张贴错误,并导致了所描述的行为。 在所有导入都正确之后,此示例在Windows和Linux上运行良好。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.