如何在 Python 的循环内将一个进程的执行与另一个进程联系起来

Question

How can I keep a loop going, while having some processes that are waiting for others inside the loop? (see the code below for clarification, it makes more sense to explain in code)

for i in range(0, len(my_list), batch_size):
    current_batch = my_list[i:i+batch_size]
    download_processes = [Popen('do stuff')] # NOT ACTUAL CODE. This downloads files.
    _ = [p.wait() for p in download_processes] # wait until all files above download before executing the code below 

    more_processes = [Popen('do stuff')] # NOT ACTUAL CODE. This zips the downloaded files
    # execute yet more processes (that process the zips files) when "more_processes" end, but keep the loop going to start downloading more files

Answer 1

1. Create a pool

2. Use loop inside pool for the batch

3. Use result = pool.map_async() with your target method

4.Do result.get(timeout) or result.wait()

5. If you get timeout or after your condition is met, return and break out of the while loop and pool.close, terminate and then join.

def process_url(url):
  # call url and process data
  
  pass

def pool_handler():
    with Pool() as p:
     for i in range(0, len(my_list), batch_size):
      current_batch_urls = my_list[i:i+batch_size]
    
      # this will create processes and process url
      r = p.map_async(process_url, current_batch_urls)
      r. wait()#wait for each batch

   #outside loop 
    p.close()
    p.join()#wait until all processes are done

if __name__ == '__main__':
    pool_handler()

Answer 2

You can use multiprocessing module to achieve this

from multiprocessing import Pool
import time, requests

urls = ["file_url1","file_url2","file_url3"]


def download_file(url):
    return requests.get(url).content.strip()

def process_url(url):
    file_content = download_file(url)
    # Process File content 

def pool_handler():
    p = Pool(2)
    p.map(process_url, urls)

if __name__ == '__main__':
    pool_handler()