Python multiprocessing queue not returning

Question

I am trying to use the multiprocessing module to load a bunch of jobs (that don't return anything - they just save to disk) and I can't seem to exit the multiprocessing Queue. I've looked up all the links on stackoverflow and google still don't understand why the jobs aren't finishing correctly.

The code seems to run fine (ie it properly queues and processes the tasks in whatever order they come - rather than assigning a list of tasks to each core). But I can't seem to figure out how to exit the queue :(

Any advice is much appreciated, I've spent hours on this.

Here's my code:

        import multiprocessing as mp

        # make arg list first
        args_in = []
        for channel in channels:
            args_in.append(...)  # append some stuff

        # make a queue and add list to queue:
        #q = mp.Queue()
        q = mp.JoinableQueue()
        for arg in args_in: 
            q.put(arg)        # add list of args

        # worker function
        def worker(q):
          for item in iter(q.get, None):
            res = cluster_channels_chunks_args(item)
            q.task_done()


        # make a list of processes and add worker function
        procs = []
        for i in range(CONFIG.resources.n_processors):
            procs.append(mp.Process(target=worker, args=(q,)))
            procs[-1].daemon = True
            procs[-1].start()

        for p in procs:
            p.join()

[EDIT] if I comment out procs[-1].daemon=True, I no longer get the correct behaviour and cores seem to be assigned a list of jobs which they must each complete. I need the cores to dynamically grab jobs out of the pool as soon as they finish (otherwise some cores finish jobs before others and I get system hangs for too long waiting for one core to finish many jobs).

Answer 1

In your last line, you are join ing daemon processes. Per Python documentation :

Note that a daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits. Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited.