如何在连续循环中使用python多处理池

Question

I am using python multiprocessing library for executing a selenium script. My code is below :

#-- start and join multiple threads ---
thread_list = []
total_threads=10 #-- no of parallel threads
for i in range(total_threads):
    t = Process(target=get_browser_and_start, args=[url,nlp,pixel])
    thread_list.append(t)
    print "starting thread..."
    t.start()

for t in thread_list:
    print "joining existing thread..."
    t.join()

As I understood the join() function, it will wait for each process to complete. But I want that as soon as a process is released, it will be assigned another task to perform new function.

It can be understood like this:

Say 8 processes started in first instance.

no_of_tasks_to_perform = 100

for i in range(no_of_tasks_to_perform):
    processes start(8)
    if process no 2 finished executing, start new process
    maintain 8 process at any point of time till 
    "i" is <= no_of_tasks_to_perform

Answer 1

Instead of starting new processes every now and then, try to put all your tasks into a multiprocessing.Queue() , and start 8 long-running processes , in each process keep accessing the task queue to get new tasks and then do the job, until there's no task any more.

In your case, it's more like this:

from multiprocessing import Queue, Process

def worker(queue):
    while not queue.empty():
        task = queue.get()

        # now start to work on your task
        get_browser_and_start(url,nlp,pixel) # url, nlp, pixel can be unpacked from task

def main():
    queue = Queue()

    # Now put tasks into queue
    no_of_tasks_to_perform = 100

    for i in range(no_of_tasks_to_perform):
        queue.put([url, nlp, pixel, ...]) 

    # Now start all processes
    process = Process(target=worker, args=(queue, ))
    process.start()
    ...
    process.join()