如何为两个不同的队列创建线程以并行运行？ -Python

Question

I have two queues for different tasks, the first crawl will start to crawl the links in the list and then it will generate more links to crawl to the queue one and also will generate new links to a different task on queue two, my program is working but the problem is: When the workers for the queue two start running it stops the workers from queue one, they are basically not running in parallel they are waiting each other finish their tasks. How can I make them run in parallel?

import threading
from queue import Queue

queue = Queue()
queue_two = Queue()

links = ['www.example.com', 'www.example.com', 'www.example.com',
         'www.example.com', 'www.example.com', 'www.example.com', 
         'www.example.com', 'www.example.com', 'www.example.com']

new_links = []

def create_workers():
    for _ in range(4):
        t = threading.Thread(target=work)
        t.daemon = True
        t.start()

    for _ in range(2):
        t = threading.Thread(target=work_two)
        t.daemon = True
        t.start()

def work():
    while True:
        work = queue.get()
        #do something
        queue.task_done()

def work_two():
    while True:
        work = queue_two.get()
        #do something
        queue_two.task_done()

def create_jobs():
    for link in links:
        queue.put(link)
    queue.join()
    crawl_two()
    crawl()

def create_jobs_two():
    for link in new_links:
        queue_two.put(link)
    queue_two.join()
    crawl_two()

def crawl():
    queued_links = links
    if len(queued_links) > 0:
        create_jobs()

def crawl_two():
    queued_links = new_links
    if len(queued_links) > 0:
        create_jobs_two()

create_workers()
crawl()

Answer 1

That is because your processing does not seem to be parallel between work and work two.

This is what happens:

1. You create workers for work and work_two
2. Crawl is called
3. Create_jobs is called - "work" workers start processing them
4. Create_jobs waits in queue.join() until all of them have completed
5. Crawl_two is called
6. Create_jobs_two is called - "work_two" workers start processing them
7. Create_jobs_two waits in queue_two.join() until all of them have completed
8. Crawl is called (start from 2. again)

Basically you never enter in a situation where work and work_two would run in parallel, as you use queue.join() to wait until all of the currently running tasks have finished. Only then will you assign tasks to the "other" queue. Your work and work_two do run in parallel within themselves, but the control structure ensures work and work_two are mutually exclusive. You need to redesign the loops and queues if you want both of them run in parallel.

You will probably also want to investigate the use of threading.Lock() to protect your global new_links variable, as I assume you will be appending things to it in your worker threads. This is absolutely fine but you need a lock to ensure two threads are not trying to do this simultaneously. But this is not related to your current problem. This only helps you avoid the next problem.

I of course do not know what you try to achieve here, but you might try solving your problem and avoiding the next one by scrapping the global new_links completely. What if your work and work_two just fed the queue of the other worker if they needed to submit tasks to them, instead of putting items into a global variable and then feeding them to the queue in the main thread?

Or you could build an "orchestration" thread that would queue tasks to workers, process responses from them and then act on that response accordingly, either queuing it back to one of the queues or accepting the result if it is "ready".