[英]Python threading multiple bash subprocesses?
您不需要線程來並行運行子流程:
from subprocess import Popen
commands = [
'date; ls -l; sleep 1; date',
'date; sleep 5; date',
'date; df -h; sleep 3; date',
'date; hostname; sleep 2; date',
'date; uname -a; date',
]
# run in parallel
processes = [Popen(cmd, shell=True) for cmd in commands]
# do other things here..
# wait for completion
for p in processes: p.wait()
要限制並發命令的數量,你可以使用multiprocessing.dummy.Pool
使用線程,並提供相同的接口multiprocessing.Pool
使用流程:
from functools import partial
from multiprocessing.dummy import Pool
from subprocess import call
pool = Pool(2) # two concurrent commands at a time
for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)):
if returncode != 0:
print("%d command failed: %d" % (i, returncode))
該答案演示了限制並發子進程數的各種技術 :它顯示了multiprocessing.Pool,concurrent.futures,線程+基於隊列的解決方案。
您可以限制並發子進程的數量,而無需使用線程/進程池:
from subprocess import Popen
from itertools import islice
max_workers = 2 # no more than 2 concurrent processes
processes = (Popen(cmd, shell=True) for cmd in commands)
running_processes = list(islice(processes, max_workers)) # start new processes
while running_processes:
for i, process in enumerate(running_processes):
if process.poll() is not None: # the process has finished
running_processes[i] = next(processes, None) # start new process
if running_processes[i] is None: # no new processes
del running_processes[i]
break
在Unix上,您可以避免繁忙的循環並在os.waitpid(-1, 0)
上進行os.waitpid(-1, 0)
,以等待任何子進程退出 。
一個簡單的線程示例:
import threading
import Queue
import commands
import time
# thread class to run a command
class ExampleThread(threading.Thread):
def __init__(self, cmd, queue):
threading.Thread.__init__(self)
self.cmd = cmd
self.queue = queue
def run(self):
# execute the command, queue the result
(status, output) = commands.getstatusoutput(self.cmd)
self.queue.put((self.cmd, output, status))
# queue where results are placed
result_queue = Queue.Queue()
# define the commands to be run in parallel, run them
cmds = ['date; ls -l; sleep 1; date',
'date; sleep 5; date',
'date; df -h; sleep 3; date',
'date; hostname; sleep 2; date',
'date; uname -a; date',
]
for cmd in cmds:
thread = ExampleThread(cmd, result_queue)
thread.start()
# print results as we get them
while threading.active_count() > 1 or not result_queue.empty():
while not result_queue.empty():
(cmd, output, status) = result_queue.get()
print('%s:' % cmd)
print(output)
print('='*60)
time.sleep(1)
請注意,有更好的方法可以執行某些操作,但這並不太復雜。 該示例為每個命令使用一個線程。 當您想要執行諸如使用有限數量的線程來處理未知數量的命令之類的事情時,復雜性開始蔓延。 掌握了線程基礎知識之后,那些更高級的技術似乎並不太復雜。 一旦掌握了這些技術,多處理將變得更加容易。
這是因為它應該執行,所以您要做的不是multithreadind,而是多處理,請參見此堆棧頁面
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.