简体   繁体   English

Python multiprocessing.Pool在Windows上很奇怪

[英]Python multiprocessing.Pool is weird on Windows

I have a super simple python script as defined here 我有一个超简单的python脚本,如这里定义

import multiprocessing
from multiprocessing import Pool

print "CPUs: " + str(multiprocessing.cpu_count())
convertPool = Pool(multiprocessing.cpu_count())

On Linux this appears to behave as I would of expected, with the program just starting and then printing out the number of cores, then exiting. 在Linux上,这似乎与我预期的一样,程序只是启动,然后打印出内核数量,然后退出。 However, on Windows this program will continue to invoke new python commands and print out "CPUs: 4" (I have 4 cores) and never stop running and will eventually kill the box. 但是,在Windows上,该程序将继续调用新的python命令并打印出“ CPU:4”(我有4个内核),并且永不停止运行,最终将杀死该计算机。 Can someone please explain what is going on here? 有人可以解释一下这是怎么回事吗?

Thanks 谢谢

EDIT: I now have the following program which still doesn't operate as I would expect 编辑:我现在有以下程序仍然无法正常运行

import sys
import os
import subprocess
from subprocess import Popen, PIPE
from threading import Thread
import multiprocessing
from multiprocessing import Pool, freeze_support

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = "posix" in sys.builtin_module_names

def myPrint(line, logWriter):
    if logWriter is None:
        # This will print without a newline, cause the process
        # has its own newlines
        sys.stdout.write(line)
    else:
        logWriter.write(line.strip())
        logWriter.write("\n")
        logWriter.flush()

# This is gotten from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
def executeCommand(cmd, logWriter):
   myPrint(cmd + "\n", logWriter)
    p = Popen(cmd, stdout=PIPE, bufsize=4096, close_fds=ON_POSIX, shell=True)
    q = Queue()
    t = Thread(target=enqueue_output, args=(p.stdout, q))
    t.daemon = True # thread dies with the program
    t.start()

    # read line without blocking
    while t.isAlive() or not q.empty():
        try:
            line = q.get_nowait() # or q.get(timeout=.1)
        except Empty:
            pass # Do nothing
        else: # If there is a line, then print it 
            if logWriter is not None:
                myPrint(line, logWriter)
        # Sleep to prevent python from using too much cpu
        time.sleep(0.05)

if __name__ == "__main__":
    freeze_support()
    convertPool = Pool(multiprocessing.cpu_count())
    # Now let's smooth and threshold all the masks
    for foo in bar:
        threshSmoothCmd = "ConvertImage.exe -i " + foo + " -o " + foo
        myPrint(threshSmoothCmd + "\n", None)
        convertPool.apply_async(executeCommand,(threshSmoothCmd, None,))

    convertPool.close()
    convertPool.join()
print "Finished processing"

ConvertImage.exe is an executable I wrote myself foo and bar are just placeholders. ConvertImage.exe是我自己写的可执行文件foo和bar只是占位符。 On Linux this will fire up multiprocessing.cpu_count() number of ConvertImage.exe processes and then print Finished processing once all of the ConvertImage.exe processes have finished. 在Linux上,这将启动multiprocessing.cpu_count()个ConvertImage.exe进程的数量,然后在所有ConvertImage.exe进程完成后打印“ Finished”处理。 On Windows this immediately fires up len(bar) ConvertImage.exe processes and then immediately prints Finished processing and exits. 在Windows上,这立即启动len(bar)ConvertImage.exe进程,然后立即打印Finished处理并退出。 How do I get the Windows version to behave like the Linux version? 如何使Windows版本的行为与Linux版本类似?

I figure this out, it was working correctly, my example program was posted wrong as I forgot some imports and that caused the described behavior. 我发现了这一点,它工作正常,由于忘记了一些导入操作,导致示例程序张贴错误,并导致了所描述的行为。 After I got all the imports correct this example worked fine on Windows and Linux. 在所有导入都正确之后,此示例在Windows和Linux上运行良好。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM