簡體   English   中英

Python multiprocessing.Pool在Windows上很奇怪

[英]Python multiprocessing.Pool is weird on Windows

我有一個超簡單的python腳本,如這里定義

import multiprocessing
from multiprocessing import Pool

print "CPUs: " + str(multiprocessing.cpu_count())
convertPool = Pool(multiprocessing.cpu_count())

在Linux上,這似乎與我預期的一樣,程序只是啟動,然后打印出內核數量,然后退出。 但是,在Windows上,該程序將繼續調用新的python命令並打印出“ CPU:4”(我有4個內核),並且永不停止運行,最終將殺死該計算機。 有人可以解釋一下這是怎么回事嗎?

謝謝

編輯:我現在有以下程序仍然無法正常運行

import sys
import os
import subprocess
from subprocess import Popen, PIPE
from threading import Thread
import multiprocessing
from multiprocessing import Pool, freeze_support

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = "posix" in sys.builtin_module_names

def myPrint(line, logWriter):
    if logWriter is None:
        # This will print without a newline, cause the process
        # has its own newlines
        sys.stdout.write(line)
    else:
        logWriter.write(line.strip())
        logWriter.write("\n")
        logWriter.flush()

# This is gotten from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
def executeCommand(cmd, logWriter):
   myPrint(cmd + "\n", logWriter)
    p = Popen(cmd, stdout=PIPE, bufsize=4096, close_fds=ON_POSIX, shell=True)
    q = Queue()
    t = Thread(target=enqueue_output, args=(p.stdout, q))
    t.daemon = True # thread dies with the program
    t.start()

    # read line without blocking
    while t.isAlive() or not q.empty():
        try:
            line = q.get_nowait() # or q.get(timeout=.1)
        except Empty:
            pass # Do nothing
        else: # If there is a line, then print it 
            if logWriter is not None:
                myPrint(line, logWriter)
        # Sleep to prevent python from using too much cpu
        time.sleep(0.05)

if __name__ == "__main__":
    freeze_support()
    convertPool = Pool(multiprocessing.cpu_count())
    # Now let's smooth and threshold all the masks
    for foo in bar:
        threshSmoothCmd = "ConvertImage.exe -i " + foo + " -o " + foo
        myPrint(threshSmoothCmd + "\n", None)
        convertPool.apply_async(executeCommand,(threshSmoothCmd, None,))

    convertPool.close()
    convertPool.join()
print "Finished processing"

ConvertImage.exe是我自己寫的可執行文件foo和bar只是占位符。 在Linux上,這將啟動multiprocessing.cpu_count()個ConvertImage.exe進程的數量,然后在所有ConvertImage.exe進程完成后打印“ Finished”處理。 在Windows上,這立即啟動len(bar)ConvertImage.exe進程,然后立即打印Finished處理並退出。 如何使Windows版本的行為與Linux版本類似?

我發現了這一點,它工作正常,由於忘記了一些導入操作,導致示例程序張貼錯誤,並導致了所描述的行為。 在所有導入都正確之后,此示例在Windows和Linux上運行良好。

暫無
暫無

聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM