In PyQT how to store data emitted from signal in QRunnable

Question

I have a process that takes awhile in a PyQT4 application using Python3. So I have been investigating using threading to speed it up. The data can easily be broken part and processed in a loop. I am using a custom QRunnable where you can pass a target function and args which is then fed into a QThreadpool . I use a pyqtSignal to emit the processed data from each worker. The Data is being received by my slot but it isn't being stored. Below is an cut down example of what I have attempted.

import sys, time
from random import uniform
from PyQt4 import QtCore, QtGui

class AWorker(QtCore.QRunnable):
    """ Generic Task for ThreadPool to execute required Kwargs =
    target (<function>): function to call
    args  (tuple): args for target
    kwargs (dict): kwargs for target """  
    def __init__(self, target=None, args=(), kwargs={}):
        super(AWorker, self).__init__()
        self.target = target 
        self.args = args
        self.kwargs = kwargs
    def run(self):
        self.target(*self.args, **self.kwargs)

class myTest(QtCore.QObject):  

    doneSignal = QtCore.pyqtSignal(int)  #Create a signal to emit the data

    def __init__(self):
        super(myTest, self).__init__()        
        self._procData = [] #Place to Store data .. maybe        
        self.pool = QtCore.QThreadPool.globalInstance()
        self.pool.setMaxThreadCount(4) #Use up to 8 threads

    def runAll(self):
        self.doneSignal.connect(self.storeData) 
        for data in range(4):
            worker = AWorker(target=self.processData, args=(data,))
            self.pool.start(worker)    

    def processData(self,data):        
        print('Crunching ...', str(data))
        outData = data+10
        time.sleep(uniform(1,3))  #Simulate this taking a random amount of time
        self.doneSignal.emit(outData)

    def storeData(self,data):        
        print('Received ...', str(data))
        self._procData.append(data)

    def getData(self):
        return self._procData

if __name__ == '__main__':
    app = QtGui.QApplication(sys.argv)
    test = myTest()
    test.runAll()
    test.pool.waitForDone()
    print('All done ... and the data is: ',test.getData())
    app.exec_()

It runs and from the print outs the following:

Crunching ... 0
Crunching ... 1
Crunching ... 3
Crunching ... 2
All done ... and the data is:  []
Received ... 13
Received ... 11
Received ... 12
Received ... 10

Ok, so the signals aren't emitted until after they all finish? maybe? or I can't assign to _procData since I am in a different thread or something? I tried removing the waitForDone() just to see but that as expected didn't help.

So what is the proper way to assemble the data output from each worker? And I suppose a follow up question, how do you ensure it is put back in the correct order.

Answer 1

I encountered a similar issue (which can also be related to this question : Emitting signals from a QRunnable ) with one of my code.
As in your case, the action in the QRunnable was executed but results weren't appended in the result list. The answer in the other post says it's because the main event loop isn't called (with something like app.exec_() ). I guess in your case it's because you try to print the returned result (which is empty) before calling app.exec_() (I might have read that events/signals from QThread or QRunnable are queued or something like that before the exec_ call ? Even if in your case as in mine, each individually result received could be printed from the callback function).

I guess a work-around could be to create a QEventLoop which will processEvents from the worker of the QThreadPool , then when it exit() all signals/events have been processed. Using your code it could look like :

def runAll(self):
    self.eventloop = QEventLoop()
    self.pool = QtCore.QThreadPool.globalInstance()
    self.pool.setMaxThreadCount(4)
    self.doneSignal.connect(self.storeData) 
    for data in range(4):
        worker = AWorker(target=self.processData, args=(data,))
        self.pool.start(worker)
    self.pool.waitForDone()
    self.eventloop.processEvents() 
    self.eventloop.exit()
    # Now your data should have been appended to your list 

(I did not tested this snippet of code but I successfully use a similar pattern for some of my QThreadPool )