Qt GUI becomes unresponsive emitting signals too fast

Question

I've got a small chat client that stores all history in an sqlite database. When the user clicks on the history tab in my application my app fetches all the relevant history and displays it in a QWebView . Im fetching from a background thread dbThread below and then sending signals to update the QWebView accordingly.

This works fine, until the database grows. When the database gets larger the app starts to almost crashes. GUI is unresponsive for a few seconds until everything is loaded (4-6 secs) depending on the database size.

I've tried to add Qt::QueuedConnection on the signals and also like mentioned above I'm handling all database queries from a background thread .

I'm guessing that I'm emitting signals too fast. Any ideas how to solve this?

Signals

connect(dbtrad, SIGNAL(addAllHistoryMessage(QString, QString, QString, QString, QString)), this, SLOT(addAllHistoryMessage(QString, QString, QString, QString, QString)), Qt::QueuedConnection);
connect(dbtrad, SIGNAL(addAllHistoryMessageInner(QString, QString, QString, QString, QString)), this, SLOT(addAllHistoryMessageInner(QString, QString, QString, QString, QString)), Qt::QueuedConnection);

Code that fetches history from the sqlite database:

// Loads all local history
void dbThread::loadAllHistory(QString agentID, QString agentName) {
    bool ret = false;
    bool retInner = false;
    QString retVal = "";

    QDateTime dateTime = dateTime.currentDateTime();
    QString dateTimeForTodayCheck = dateTime.toString("yyyy-MM-dd");

    if (db.isOpen()) {
        QSqlQuery query(db);
        QSqlQuery queryInner(db);

        ret = query.exec(QString("SELECT channelID, sender, time, message from chatHistory WHERE sender != 'ServerMessage' AND channelID NOT LIKE '%Agent%' GROUP BY channelID order by time DESC;"));

        if (ret) {

            while (query.next()) {    
                QString channelID = query.value(0).toString();
                QString sender = query.value(1).toString();
                QString time = query.value(2).toString();
                QString msg = query.value(3).toString();


                QString timeStr;
                QString fmt = "yyyy-MM-dd hh:mm:ss";
                QDateTime dt = QDateTime::fromString(time, fmt);
                QDateTime dtCompare = QDateTime::fromString(time, fmt);

                if(dateTimeForTodayCheck == dtCompare.toString("yyyy-MM-dd")) { // If today
                    timeStr = "Today " + dt.toString("hh:mm");
                } else {
                    timeStr = dt.toString("dd MMM yyyy");
                }

                if(sender == agentID) {
                    sender = agentName;
                }
                // Grab all the tags
                QString tempTagsForChannelID = getHistoryTagsString(channelID);

                emit addAllHistoryMessage(channelID, sender, timeStr, msg, tempTagsForChannelID);

                // Load sub-history
                retInner = queryInner.exec(QString("SELECT * from chatHistory WHERE sender != 'ServerMessage' AND channelID = '%1' and message != '%2' order by time DESC;").arg(channelID).arg(msg));

                if (retInner) {
                    while (queryInner.next()) {
                        QString channelIDInner = queryInner.value(0).toString();
                        QString senderInner = queryInner.value(1).toString();
                        QString timeInner = queryInner.value(4).toString();
                        QString msgInner = queryInner.value(2).toString();

                        QString timeStr2;
                        QString fmt = "yyyy-MM-dd hh:mm:ss";
                        QDateTime dt = QDateTime::fromString(timeInner, fmt);
                        QDateTime dtCompare = QDateTime::fromString(timeInner, fmt);

                        if(dateTimeForTodayCheck == dtCompare.toString("yyyy-MM-dd")) { // If today
                            timeStr2 = "Today " + dt.toString("hh:mm");
                        } else {
                            timeStr2 = dt.toString("dd MMM yyyy");
                        }

                        if(senderInner == agentID) {
                            senderInner = agentName;
                        }

                        emit addAllHistoryMessageInner(channelIDInner, senderInner, timeStr2, msgInner, tempTagsForChannelID);
                    }
                }
            }
        }
    }
}

My code to update:

void MainWindow::addAllHistoryMessageInner(QString channelIDInner, QString senderInner, QString timeStr2, QString msgInner, QString tempTagsForChannelID) {
    ui->webViewHistory->page()->mainFrame()->evaluateJavaScript("$('#history tbody').append('<tr id=\"" + channelIDInner+ "\" class=\"hiddenRow\"><td>" + senderInner + "</td><td align=\"center\">" + timeStr2 + "</td><td align=\"center\" style=\"word-wrap:break-word;\">" + msgInner.remove(QRegExp("<[^>]*>")) + "</td><td align=\"center\">" + tempTagsForChannelID + "</td></tr>');undefined");
}

void MainWindow::addAllHistoryMessage(QString channelID, QString sender, QString timeStr, QString msg, QString tempTagsForChannelID) {
    ui->webViewHistory->page()->mainFrame()->evaluateJavaScript("$('#history tbody').append('<tr id=\"" + channelID + "\"><td>" + sender + "</td><td align=\"center\">" + timeStr + "</td><td align=\"center\" style=\"word-wrap:break-word;\">" + msg.remove(QRegExp("<[^>]*>")) + "</td><td align=\"center\" style=\"word-wrap:break-word;\">" + tempTagsForChannelID + "</td></tr>');undefined");
}

Edit: Implementation of dbThread

thread = new QThread(this);
dbtrad = new dbThread();
dbtrad->moveToThread(thread);

Edit 2: This is how I call loadAllHistory

I create a signal:

connect(this, SIGNAL(loadAllHistoryS(QString, QString)), dbtrad, SLOT(loadAllHistory(QString, QString)));

And call it like this:

emit loadAllHistoryS(agentID, agentName);

Answer 1

The problem is, that your main thread is interrupted for every single row in the innerQuery . This destroys the benefits of loading the data in separate thread. Probably the overhead of the signal/slot communication over thread boundaries is even higher than the costs of loading a single row from the database.

I would recommend to collect the rows in a QList instance the while loop. When done, push the complete result via one signal invocation to the main thread:

First declare a simple class for storing history items:

class HistoryItem {
    public:
        QString channelID;
        /* additonal fields omitted for brevity */
        /* also, private fields with getters and setters would be better */
}

Then, create a list of such objects before the while loop:

QList<HistoryItem*> innerResult;
while (queryInner.next()) {
      /* snip */
      HistoryItem* item = new HistoryItem();
      item.channelId = channelIDInner;
      /* more lines ommited */
      innerResult.append(historyItem);
}
emit historyLoaded(innerResult);

Obviously, you also need a matching signal definition in your worker class:

Q_SIGNAL void historyLoaded(QList<HistoryItem*> result);    

---

Also as noted in the comments, you have to start the background thread with QThread::start() .

Answer 2

You may benefit from continuous loading:

You only load the elements that will be in view and only request the next set when they get scrolled into view (or just at a much lower pace in the background in general).

This can for example be done with Adding an object to the window object with the appropriate signal and letting the js trigger it when the end becomes visible.