Python Creation of a semaphore or a check after finishing the execution of a list of threads

Question

I am running a script that needs to run many external commands. And I have several blocks of commands that needs to be run in parallel, together.

So, I need to control that this pool of threads have finished to start the next one.

The commands to be run:

import pathlib
def thread_command(path_files, command):
    _path_files= pathlib.Path(path_files)
    if pathlib.Path(path_files).exists():
    for file in _path_files.glob('**/*.ext'):

        ext_file = Ext(file, path_files, tmp_rules)
        if threads == None:
            threads.insert(0, threading.Thread(target=command))
        else:
            threads.append(threading.Thread(target=command))
    return threads

#Threads running
command1 = ext_file.command1
command2= ext_file.command2
threads1 = thread_command(pathToFiles, command1)
for thread in threads:
   thread.daemon = True
   thread.start()
 do_other_things()
 threads2 = thread_command(pathToFiles, command2)
 for thread in threads2:
    thread.daemon = True
    thread.start()

I would need to find a way or a procedure to control when the first list of threads finish to continue with the execution of the program. In summary, I would like to run the whole list threads, and wait that all the threads run there finish, and then start the second pool of threads.

Thank you for the help, in advance.

Answer 1

Don't use 'daemon' threads, instead wait for them to join back to the main thread, eg:

commands = [ext_file.command1, ext_file.command2]  # etc.
for command in commands:  # loop through commands in succession
    # get the thread list and start it immediately
    threads_list = [t for t in thread_command(pathToFiles, command) if t.start() is None]
    for t in threads_list:  # loop through all started threads and...
        t.join()  # ... wait for them to join back

But your thread_command() function makes little sense on its own - for starters, you'll get an error the moment you try to insert/append the threads list as it's not defined (at least not within the function context). Second, why would you go over the file list again and again for each command, unless you're expecting for the file list to change?