Multiprocessing script gets stuck

Question

I have the following Python code:

def workPackage(args):
   try:      
    outputdata                  = dict()
    iterator                        = 1
    for name in outputnames:
        outputdata[name]            = []
    for filename in filelist:
        read_data                   = np.genfromtxt(filename, comments="#", unpack=True, names=datacolnames, delimiter=";")
        mean_va1                    = np.mean(read_data["val1"])
        mean_va2                    = np.mean(read_data["val2"])
        outputdata[outputnames[0]].append(read_data["setpoint"][0])
        outputdata[outputnames[1]].append(mean_val1)
        outputdata[outputnames[2]].append(mean_val2)        
        outputdata[outputnames[3]].append(mean_val1-mean_val2)
        outputdata[outputnames[4]].append((mean_val1-mean_val2)/read_data["setpoint"][0]*100)
        outputdata[outputnames[5]].append(2*np.std(read_data["val1"]))
        outputdata[outputnames[6]].append(2*np.std(read_data["val2"]))      


        print("Process "+str(identifier+1)+": "+str(round(100*(iterator/len(filelist)),1))+"% complete")
        iterator    = iterator+1

    queue.put (outputdata)
 except:
 some message

if __name__ == '__main__':
"Main script"

This code is used to evaluate a large amount of measurement data. In total I got some 900 files across multiple directories (about 13GB in total). The main script determines all the filepaths and stores them in 4 chunks. Each chunk (list of filepaths) is given to one process.

    try:
      print("Distributing the workload on "+str(numberOfProcesses)+" processes...")                     
      for i in range(0,numberOfProcesses):
        q[i]                = multiprocessing.Queue()
        Processes[i]        = multiprocessing.Process(target=workPackage, args=(filelistChunks[i], colnames, outputdatanames, i, q[i]))
        Processes[i].start()
      for i in range(0,numberOfProcesses):
        Processes[i].join()
    except:
       print("Exception while processing stuff...")

After that the restuls are read from the queue and stored to an output file. Now here's my problem: The script starts the 4 processes and each of them runs to 100% (see the print in the workPackage function). They don't finish at the same time but within about 2 minutes. But then the script simply stops. If I limit the amount of data to process by simply cutting the filelist it sometimes runs until the end but sometimes doesn't. I don't get, why the script simply gets stuck after all processes reach 100%.

I seriously don't know what's happening there.

Answer 1

You add items to the queue with queue.put(), then call queue.join(), but I don't see where you call queue.get() or queue.task_done(). Join won't release the thread until the queue is empty and task_done() has been called on each item.