[英]Python run to run Batch Command in Parralel
I have a List which has all my Batch Command.(More than 1000) count I need to run the commands from this list 5 at a time(in Parallel), If any one completes the 6th should kick off. 我有一个列表,其中包含我的所有批处理命令。(超过1000个)计数我需要一次(并行)运行此列表5中的命令,如果任何一个完成,则第6个命令应该启动。
Can you please assist? 你能帮忙吗? Thanks
谢谢
If you don't need output from the Batch commands, you can simply 如果不需要Batch命令的输出,则只需
import subprocess
subprocess.Popen("command string here", shell=True)
This will run your batch code in the shell attached to this Python runtime. 这将在此Python运行时附带的shell中运行批处理代码。
To run things in parallel, you can just track how many are currently running. 要并行运行,您只需跟踪当前正在运行的数量即可。 I like to use threads for this
我喜欢为此使用线程
import subprocess
from threading import Thread
import time
processing = 0
def call_batch(command):
global processing
process = subprocess.Popen("command string here", shell=True)
process.wait()
processing -= 1
if __name__ == "__main__":
commands = []
##load commands
for command in commands:
if processing < 5:
t = Thread(target=call_batch, args=(command))
t.daemon = True
t.start()
processing += 1
else:
time.sleep(0.1) # I don't know how long you expect these commands to take so this is allowing a max of 5 * (1/0.1) = 50 per second
If you come from another programming background, you will notice the lack of locks. 如果您来自其他编程背景,则会注意到缺少锁。 This is because of the global interpreter lock.
这是因为全局解释器锁定。
If you know a lot about Python, you will notice my recommendation to use shell=True
. 如果您对Python了解很多,您会注意到我的建议使用
shell=True
。 I recommend it because it is simple and not dangerous when executed on trusted input however the OP should decide whether to use shell=True
based on the scenario. 我建议这样做,因为它在受信任的输入上执行时很简单并且没有危险,但是OP应该根据情况决定是否使用
shell=True
。
Reading on Thread
: https://docs.python.org/2/library/threading.html#thread-objects 在
Thread
阅读: https : //docs.python.org/2/library/threading.html#thread-objects
Reading on subprocess
: https://docs.python.org/2/library/subprocess.html 阅读
subprocess
: https : //docs.python.org/2/library/subprocess.html
Docs on why shell=True is dangerous
: https://docs.python.org/2/library/subprocess.html#frequently-used-arguments 关于
shell=True is dangerous
为何shell=True is dangerous
文档: https : //docs.python.org/2/library/subprocess.html#frequently-used-arguments
You may like to use the ThreadPool
from the multiprocessing
library if you don't need a lot of control over the system. 如果您不需要对系统进行大量控制,则可以使用
multiprocessing
库中的ThreadPool
。 See http://chriskiehl.com/article/parallelism-in-one-line/ about half-way down for an example of multiprocessing.ThreadPool.map
for mapping a function to a number of threads. 有关将函数映射到多个线程的
multiprocessing.ThreadPool.map
的示例,请参阅http://chriskiehl.com/article/parallelism-in-one-line/大约一半。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.