[英]subprocess.wait() not waiting for Popen process to finish (when using threads)?
I am experiencing some problems when using subprocess.Popen()
to spawn several instances of the same application from my python script using threads to have them running simultaneously.在使用
subprocess.Popen()
从我的 python 脚本中使用线程让它们同时运行时,我遇到了一些问题。 In each thread I run the application using the popen()
call, and then I wait for it to finish by calling wait()
.在每个线程中,我使用
popen()
调用运行应用程序,然后通过调用wait()
等待它完成。 The problem seems to be that the wait()
-call does not actually wait for the process to finish.问题似乎是
wait()
调用实际上并没有等待进程完成。 I experimented by using only one thread, and by printing out text messages when the process starts, and when it finishes.我尝试只使用一个线程,并在进程开始和结束时打印出文本消息。 So the thread function would look something like this:
所以线程 function 看起来像这样:
def worker():
while True:
job = q.get() # q is a global Queue of jobs
print('Starting process %d' % job['id'])
proc = subprocess.Popen(job['cmd'], shell=True)
proc.wait()
print('Finished process %d' % job['id'])
job.task_done()
But even when I only use one thread, it will print out several "Starting process..." messages, before any "Finished process..." message appears.但即使我只使用一个线程,它也会在出现任何“已完成进程...”消息之前打印出几条“正在启动进程...”消息。 Are there any cases when
wait()
does not actually wait?是否存在
wait()
实际上不等待的情况? I have several different external applications (C++ console applications), which in turn will have several instances running simultaneously, and for some of them my code works, but for others it won't.我有几个不同的外部应用程序(C++ 控制台应用程序),它们又会同时运行多个实例,对于其中一些,我的代码可以工作,但对于其他一些则不行。 Could there be some issue with the external applications that somehow affects the call to
wait()
?外部应用程序是否存在某些问题会以某种方式影响对
wait()
的调用? The code for creating the threads looks something like this:创建线程的代码如下所示:
for i in range(1):
t = Thread(target=worker)
t.daemon = True
t.start()
q.join() # Wait for the queue to empty
Update 1 : I should also add that for some of the external applications I sometimes get a return code ( proc.returncode
) of -1073471801.更新 1 :我还应该补充一点,对于某些外部应用程序,我有时会得到 -1073471801 的返回码 (
proc.returncode
)。 For example, one of the external applications will give that return code the first two times Popen
is called, but not the last two (when I have four jobs).例如,其中一个外部应用程序将在前两次调用
Popen
时给出该返回码,但不是最后两次(当我有四个作业时)。
Update 2 : To clear things up, right now I have four jobs in the queue, which are four different test cases.更新 2 :为了解决问题,现在我在队列中有四个作业,它们是四个不同的测试用例。 When I run my code, for one of the external applications the first two
Popen
-calls generate the return code -1073471801.当我运行我的代码时,对于其中一个外部应用程序,前两个
Popen
调用会生成返回码 -1073471801。 But if I print the exact command which Popen
calls, and run it in a command window, it executes without any problems.但是,如果我打印
Popen
调用的确切命令,并在命令 window 中运行它,它就会毫无问题地执行。
Solved!解决了! I managed to solve the issues I was having.
我设法解决了我遇到的问题。 I think the problem was my lack of experience in threaded programming.
我认为问题在于我缺乏线程编程方面的经验。 I missed the fact that when I had created my first worker threads, they would keep on living until the python script exits.
我错过了这样一个事实,即当我创建我的第一个工作线程时,它们会继续存在,直到 python 脚本退出。 By mistake I created more worker threads each time I put new items on the queue (I do that in batches for every external program I want to run).
每次我将新项目放入队列时,我都会错误地创建更多工作线程(我为每个要运行的外部程序分批执行此操作)。 So by the time I got to the fourth external application, I had four threads running simultaneously even though I only thought I had one.
所以当我到达第四个外部应用程序时,我有四个线程同时运行,尽管我只以为我有一个。
You could also use check_call()
instead of Popen.您也可以使用
check_call()
代替 Popen。 check_call()
waits for the command to finish, even when shell=True
and then returns the exit code of the job. check_call()
等待命令完成,即使shell=True
然后返回作业的退出代码。
Sadly when running your subprocess using shell=True
, wait(
) will only wait for the sh
subprocess to finish and not for the command cmd
.遗憾的是,当使用
shell=True
运行子进程时, wait(
)只会等待sh
子进程完成,而不是等待命令cmd
。
I will suggest if it possible to don't use the shell=True
, if not possible you can create a process group like in this answer and use os.waitpid to wait for the process group not just the shell process.我会建议是否可以不使用
shell=True
,如果不可能,您可以像在这个答案中那样创建一个进程组并使用os.waitpid来等待进程组,而不仅仅是 shell 进程。
Hope this was helpful:)希望这有帮助:)
I was having issues as well, but was inspired by yours.我也遇到了问题,但受到了你的启发。
Mine looks like this, and works beautifully:我的看起来像这样,并且工作得很好:
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags = subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
proc = subprocess.Popen(command, startupinfo=startupinfo)
proc.communicate()
proc.wait()
Notice that this one hides the window as well.请注意,这个也隐藏了 window。
Make sure all applications your are calling have valid system return codes when they finish确保您调用的所有应用程序在完成时都具有有效的系统返回码
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.