[英]Python3 subprocess in multiprocessing
I'm a beginner in writing concurrent code. 我是编写并发代码的初学者。
I'm writing a code that takes the users ID and tries to return a full name of the user, the query takes a second or so to execute, so I was hoping to involve multiprocessing to collect the data faster; 我正在编写一个使用用户ID并尝试返回用户全名的代码,执行该查询需要一秒钟左右的时间,因此我希望可以通过多处理来更快地收集数据。 I think I'm close, but I don't get how the framework needs to be implemented correctly.
我想我已经接近了,但是我不了解如何正确实施框架。
from subprocess import getoutput
from multiprocessing import Pool
all_users = ['User1', 'User2', 'User3', 'User4', 'User5', 'User6'] # example list
def get_name(userid):
name = getoutput('net users {} /domain | findstr "full name:"'.format(userid)).replace('Full Name', '').strip().split('\n')[0]
return {userid : name}
if __name__ == '__main__':
with Pool(4) as p:
print(p.map(get_name, all_users))
print(' --------- finished')
print(' - exiting - '))
This is just a single step in a multi-step script; 这只是多步骤脚本中的一个步骤。 and the output appears as follows: (ignore the "the user name could not be found" part, just an example)
并且输出显示如下:(忽略“找不到用户名”部分,仅作为示例)
- exiting -
- exiting -
- exiting -
- exiting -
[{'User1': 'The user name could not be found.'}, {'User2': 'The user name could not be found.'}, {'User3': 'The user name could not be found.'}, {'User4': 'The user name could not be found.'}, {'User5': 'The user name could not be found.'}, {'User6': 'The user name could not be found.'}]
--------- finished
- exiting -
I'm trying to structure the program as follows: 我正在尝试构建程序,如下所示:
I've tried reading on the subject from various sources, but I just can't grasp the structure somehow... as I understood, I'm getting four - exiting - statements at the beginning as I have 4 cores, but how do I encapsulate this part of the code so that while it's running, nothing else is happening and the - exiting - is written only once at the end of it. 我尝试从各种来源阅读该主题,但我只是无法以某种方式掌握其结构...据我所知,由于我有4个核心,因此一开始我会遇到四个-退出-语句,但是该怎么做我封装了这部分代码,以便在其运行时没有任何其他事情发生,并且-退出-在其末尾仅编写一次。
You need to use a pool.close() statement in your with loop: 您需要在with循环中使用pool.close()语句:
with Pool(4) as p:
print(p.map(get_name, all_users))
p.close()
Josh Hayes already gave the right answer. Josh Hayes已经给出了正确的答案。 If you use pool like that, it is going to call
terminate
on exit ( https://docs.python.org/3.4/library/multiprocessing.html?highlight=process ) since Python 3.3. 如果您使用那样的池,它将自Python 3.3开始在退出时调用
terminate
( https://docs.python.org/3.4/library/multiprocessing.html?highlight=process )。 You have to add p.close()
to properly finish. 您必须添加
p.close()
才能正确完成。 However, your last bracket is too much and you should not see more than one finished
and exiting
print because those calls are not within the pool. 但是,您的最后一个括号太多了,您应该看不到一个以上的
finished
并exiting
打印,因为这些调用不在池中。 How do you start your script? 您如何启动脚本? Which Python version are you using?
您正在使用哪个Python版本?
Edit: You may try to add: 编辑:您可以尝试添加:
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
all_users = ['User1', 'User2', 'User3', 'User4', 'User5', 'User6'] # example list
def get_name(userid):
name = getoutput('net users {} /domain | findstr "full name:"'.format(userid)).replace('Full Name', '').strip().split('\n')[0]
print(info("p "))
return {userid : name}
and call info("whatever")
instead of exiting
and see which processes are at work here. 并致电
info("whatever")
而不是exiting
并查看此处正在运行的进程。 Which OS are you using? 您正在使用哪个操作系统? At least on Linux it makes sense.
至少在Linux上是有意义的。
Similar question was answered in the following link: multiple output returned from python multiprocessing function 以下链接回答了类似的问题: python multiprocessing函数返回了多个输出
To summarize: 总结一下:
# Import stuff
#If the worker code is in the main thread, exclude it from the if statement:
def worker():
#worker code
if __name__ == '__main__':
#execute whatever you want, it will only be executed
#as often as you intend it to
#execute the function that starts multiprocessing,
#All code outside of the if statement will be executed multiple times
#depending on the # of assigned worker threads.
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.