How to handle loading all records of a big collection and listening for new data on mongodb atlas?

Question

I have an angular app that when it loads gets all the data from a mongodb atlas collection that has around 12K records it is increasing every day, so a large data set. This data is loaded into a table from primeng. We want all data available at all times in the table so we can query see and query the whole data set in memory. I have two main problems:

1. Since the data set is large, it takes about 10 seconds right now to load, and obviously will increase as we get more data. So i want to speed that up

2. The data is being dumped into mondogb atlas from an external source. Once I load the data in the website, i want to be able to listen for new data coming in and update my UI without the need to reload the page because of point 1) i made above. How can i do that?

I've looked into different topics online things like redis with mongodb, also saw this answer that talks about watching for changes on a mongodb collection: How to listen for changes to a MongoDB collection?

What I'm having a hard time with is how to implement it all together, since I am a newbie in angular and this topic of handling large data sets efficiently.

Below is a snippet of the code that loads the data into the table when the page loads.

 ngOnInit() { this.selectedColumns = this.cols; this.loading = true; this.handleEventData(); this.setCustomFilters(); } handleEventData = async function() { this.eventService.getEvents().subscribe((data: any[]) => { this.events = this.flattenEventData(data); this.makeEventDataUserFriendly(this.events); this.setDistinctOptions(this.events); this.loading = false; }) }

 <p-table [value]="events" #dt [columns]="selectedColumns" [paginator]="true" [rows]="250" [loading]="loading" [autoLayout]=true [reorderableColumns]="true" [rowsPerPageOptions]="['250','500','1000']" dataKey="_id" rowExpandMode="single" selectionMode="single">

And here's the code that gets called when the this.eventService.getEvents() executes. fieldsToIgnore has a list of fields I don't want to include in the result. I'm using monk as my db driver:

 exports.GetEvents = function() { return db.events .find({}, { sort: { createdAt: -1 }, fields: fieldsToIgnore }) .then(docs => docs) .catch(); };

Answer 1

you implemintation of p-table

<p-table [value]="events" #dt [columns]="selectedColumns" [paginator]="true" [rows]="250" [loading]="loading"
  [autoLayout]=true [reorderableColumns]="true" [rowsPerPageOptions]="['250','500','1000']" dataKey="_id"
  rowExpandMode="single" selectionMode="single">

neet to request all the data and this is not efficient and will impact the performance of you all or will crash the browser.

primeng table support Lazy mode is handy to deal with large datasets, instead of loading the entire data, small chunks of data is loaded by invoking onLazyLoad callback everytime paging, sorting and filtering happens.

👉 table lazy mode

if you do want to see all the data as once for any resone current version of primeng support virtual scroll so you it will look like display all the data but it will get the data from database has chunks but during scrolling so it will give a better user experience

check the example of 👉 loading 250k of data