Starting processes in threads as spawned

Question

I have a problem where I am looking to create a processing class, and I want to feed data into it, then put it into a thread, and tell it to start working. Currently, this is what I have to do that:

ProcessingClass *worker = new ProcessingClass(myData);
connect(this, SIGNAL(startWorking()), worker, SLOT(startWorking()));
connect(worker, SIGNAL(doneWorking(data)), this, SLOT(workerFinished(data)));
QThread *workerThread = new QThread;
worker->moveToThread(workerThread);
workerThread->start();

emit(startWorking());

What this ends up doing is it creates the worker, gives it the data that I want it to process, connects what needs to be connected for everything to work, moves it on over to its thread, and then emits the signal to start working. This more or less does what I want it to, but there is an issue. Namely, I want to be putting this into a loop:

while (reason){
    ...//gathering data to give to the worker
    ProcessingClass *worker = new ProcessingClass(myData);
    connect(this, SIGNAL(startWorking()), worker, SLOT(startWorking()));
    connect(worker, SIGNAL(doneWorking(data)), this, SLOT(workerFinished(data)));
    QThread *workerThread = new QThread;
    worker->moveToThread(workerThread);
    workerThread->start();
}
emit(startWorking());

This does accomplish what I want it to do, but it does so by sticking everything in memory, waiting until it is all there, and then set off every single thread simultaneously to compete for resources until they are all done. Considering that for my current data amount this is over 1000 different threads, each of which takes (from previous iterations of this program) ~1-2 minutes to process the information, and the last version crashed because it ran out of memory (I think...) I don't particularly like this method much anymore.

What I would like to do is figure out a way to move the resources that I want to the thread, and then set off the thread to do the work immediately. Then I want to be able to pause the loop after some amount of threads are running (so as to not overload the computer again) and then continue this loop and set off the next thread after one of the previous threads is done.

Is there a nicer way to accomplish this?

Answer 1

You should have a fixed number of worker threads, and iterate the loop only when there are threads that are not busy.

If you insist on using a QObject , you can create a QRunnable wrapper to run the worker objects until completion in a thread pool, and to track their progress to issue more work:

class ProcessingRunnable : public ProcessingClass, public QRunnable {
  void run() Q_DECL_OVERRIDE {
    QEventLoop loop;
    moveToThread(QThread::currentThread());
    QMetaObject::invokeMethod(this, "startWorking", Qt::QueuedConnection);
    loop.exec();
    moveToThread(0);
  }
public:
  explicit ProcessingRunnable(const Data & data) : 
    ProcessingClass(data) {
    setAutoDelete(false);
    moveToThread(0); // we will be moved to a worker thread later
  }
};

class JobManager : public QObject {
  Q_OBJECT
  QThreadPool m_pool;
  QScopedPointer<ProcessingRunnable> m_worker;
  int m_jobs;
public:
  Q_SIGNAL void allJobsFinished();
  Q_SLOT void runJobs() {
    while (true) {
      if (m_worker) {
        if (m_pool.tryStart(m_worker.data())
          m_worker.take();
        else
          break;
        }
      }
      if (! reason) break;
      ... // gather data to give to the worker
      m_worker.reset(new ProcessingRunnable(myData));
      ++ m_jobs;
      connect(m_worker, &ProcessingRunnable::doneWorking, [this]{
        -- m_jobs;
        runJobs();
        if (! m_jobs) emit allJobsFinished();
      });
    }
  }
  explicit JobManager(QObject * parent = 0) : QObject(parent),
    m_jobs(0) {
  }
}

int main(int argc, char ** argv) {
  QCoreApplication app(argc, argv);
  ...
  JobManager manager;
  QObject::connect(&manager, &JobManger::allJobsFinished,
                   &app, &QCoreApplication::quit, Qt::QueuedConnection);
  manager.runJobs();
  ...
  return app.exec();
}

For this kind of an application, it might be simpler to make your ProcessingClass be a simple functor, not a QObject , and use QtConcurrent::Run and QFutureWatcher .