Meteor js: The page taking too much time to query with mongodb if records are around 100000

Question

I am working on meteor js, there will be huge data in mongodb database. For now it is around 50000 messages in database. I am providing the code that I am currently using. For some reason the application is taking too much time to render or load the data from database. Also one more thing, if I am doing any activity,eg just like the messages the app fetches the messages again from database.

Template.messages.helper({
    linkMessages() {
        var ids = _.pluck(Meteor.user().subscription, '_id');
        var messages = Messages.find({ $or: [{ feedId: { $exists: false }, link: { $exists: true } }, { feedId: { $in: ids }, link: { $exists: true } }] }, { sort: { timestamp: 1 }, limit: Session.get("linkMessageLimit") }).fetch();
        return messages;
    }

})

calling publication in oncreate method

Template.roomView.onCreated(function() {
    const self = this;
    Deps.autorun(function() {
        Meteor.subscribe('messages', Session.get('currentRoom'), Session.get('textMessageLimit'), {
            onReady() {
                isReady.messages = true;
                if (scroll.needScroll) {
                    scroll.needScroll = false;
                    if (scroll.previousMessage) {
                        Client.scrollToMessageText(scroll.previousMessage);
                    }
                } else {
                    Meteor.setTimeout(function() {
                        Client.scrollChatToBottomMsg();
                    }, 1000)
                }
            }
        });

    });
});`

The publication function on server:

Meteor.publish('messages', function(roomId, limit) {
    check(roomId, String);
    check(limit, Match.Integer);

        let query = { $or: [{ link: {$exists: false} }, { feedId: { $exists: false } }] };

        const thresholdMessage = Messages.findOne(query, { skip: limit, sort: { timestamp: 1 } });

        if (thresholdMessage && thresholdMessage.timestamp) {
            query.timestamp = { $gt: thresholdMessage.timestamp };
        }

        return Messages.find(query, { sort: { timestamp: -1 } });

});

Answer 1

It is not a good practice to allow mini-mongodb to get populated with such a huge data. Though Meteor JS is good at this too, still it will take some amount of time taking into consideration the network traffic, bandwidth etc.

Irrespective of whether it is unlimited scroll or simple pagination I would suggest you to use pagination. I have already got it accepted and it works like charm, here is the answer and entire code for pagination .

My pagination solution is server specific, so it performs good. Collection publish is limited to the limit provided from subscription.

NOTE: There is yet no such proper full fledged solution for table with search and pagination and much more facility which makes it very flexible as per our need. I suggest to create your own.

MORE INSIGHTS:

https://www.quora.com/Does-Meteor-work-well-with-large-datasets

https://forums.meteor.com/t/very-big-data-collection-in-mongodb-how-to-fetch-in-meteor-js/6571/7

https://projectricochet.com/blog/top-10-meteor-performance-problems