[英]Multiprocessing in django and Python code
I am trying to implement multiprocessing in my application on windows system.我正在尝试在 windows 系统上的应用程序中实现多处理。
The scenario is: From GUI, when i click "Run" button control comes to a python function(which is not a main function).场景是:从 GUI 中,当我单击“运行”按钮时,控件会出现 python 函数(这不是主要函数)。
Now in this function I am running loop and reading/executing multiple file one at a time.现在在这个 function 中,我正在运行循环并一次读取/执行多个文件。 I want this to happen in parallel.我希望这同时发生。
But as multiprocessing.process() need __name__ ='__main__'
, my function mentioned in " target = function name
" in multiprocessing() is not being invoked.但是由于 multiprocessing.process() 需要__name__ ='__main__'
,我的 function 在 multiprocessing() 的“ target = function name
”中提到的没有被调用。
How can I make it happen.我怎样才能让它发生。 If multiprocessing seems wrong way then any alternative way to improve code performance?如果 multiprocessing 似乎是错误的方式,那么还有其他方法可以提高代码性能吗?
Adding Sample code(please note that this is just a psudo code where i have added high level code to understand the flow, please excuse any syntax error):添加示例代码(请注意,这只是一个伪代码,我在其中添加了高级代码以理解流程,请原谅任何语法错误):
urls.py file: urls.py 文件:
from django.urls import path
from textapp import views
urlpatterns = [
path('execute/',views.functiontomultiprocess),
...
other urls
]
views.py:意见.py:
def functiontomultiprocess(request):
nprocess = []
for doc in alldocs:
p = multiprocess.Process(function2)
p.start() # start process
nprocess.append(p)
for p1 in nprocess:
p1.join()
Task runner can use, in particular Celery . Task runner 可以使用,特别是Celery 。
By means of Celery it is possible to create "turn of tasks":通过 Celery 可以创建“任务轮”:
my_task.py我的任务.py
from celery import task
@task
def myJob(*args,**kwargs):
# main task
# . . .
my_views.py我的观点.py
from django.shortcuts import render_to_response as rtr
from .tasks import myJob
def view(request):
# view
# . . .
myJob.delay(*args,**kwargs)
return rtr('template.html', {'message': 'Job has been entered'})
The call of .delay will register * myJob * for performance by one of yours * celery *, but won't block representation performance. .delay 的调用将注册 * myJob * 以供您的一个 * celery * 执行,但不会阻止表示执行。
The task isn't carried out until the worker doesn't become free therefore you should have no problems with number of processes.直到工作人员没有空闲时才会执行任务,因此您应该不会遇到进程数问题。
This is too long to specify in a comment, so:这太长了,无法在评论中指定,因此:
Again, I have no expertise in Django, but I would think this would not cause a problem on either Windows or Linux/Unix.同样,我没有 Django 方面的专业知识,但我认为这不会在 Windows 或 Linux/Unix 上造成问题。 However, you did not specify your platform, which was requested.但是,您没有指定所要求的平台。 But moreover, the code you provided would accomplish very little because your loop creates a process and waits for it to complete before creating the next process.但此外,您提供的代码将完成很少的工作,因为您的循环会创建一个进程并等待它完成,然后再创建下一个进程。 In the end you never have more than one process running at a time and thus there is no parallelism .最后,您永远不会同时运行一个以上的进程,因此没有并行性。 To correct that, try the following:要更正该问题,请尝试以下操作:
def functiontomultiprocess(request):
processes = []
for doc in alldocs: # where is alldocs defined?
p = multiprocess.Process(function2, args=(doc,)) # pass doc to function2
processess.append(p)
p.start()
# now wait for the processes to complete
for p in processes:
p.join()
Or if you want to use a pool, you have choices.或者,如果您想使用游泳池,您可以选择。 This uses the concurrent.futures
module:这使用了concurrent.futures
模块:
import concurrent.futures
def functiontomultiprocess(request):
"""
Does it make sense to create more processes than CPUs you have?
It might if there is a lot of I/O. In which case try:
n_processes = len(alldocs)
"""
n_processes = min(len(alldocs), multiprocessing.cpu_count())
with concurrent.futures.ProcessPoolExecutor(max_workers=n_processes) as executor:
futures = [executor.submit(function2, doc) for doc in alldocs] # create sub-processes
return_values = [future.result() for future in futures] # get return values from function2
This uses the multiprocessing
module:这使用multiprocessing
模块:
import multiprocessing
def functiontomultiprocess(request):
n_processes = min(len(alldocs), multiprocessing.cpu_count())
with multiprocessing.Pool(processes=n_processes) as pool:
results = [pool.apply_async(function2, (doc,)) for doc in alldocs] # create sub-processes
return_values = [result.get() for result in results] # get return values from function2
Now you just have to try it and see.现在你只需要尝试看看。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.