How to load “raw” row data into ag-grid

Question

I'm dealing with an high throughput problem. My goal is to display, at least on a chrome browser, a grid composed by 1M of rows.

These rows are dynamically fetched from a python server running on the same machine. This server has already loaded the whole dataset in memory. The communications between client (the browser) and server (python) take place through websocket. The grid has the option virtualPaging: true .

So far I reach some good performances loading pages of 100 rows each. Despite that, loading the whole 1M dataset at the beginning (therefore without the remote fetching of rows) , shows significant improvement in scrolling (no "white rows" effect).

I want to achieve the same performance without storing in the browser memory the whole dataset.

The first step that I would try is to avoid some conversions steps. The client receives from the server an array of arrays, this means that the row model on the server is "positional" (given r as a generic row, r[0] is the element related to first column, r[1] to the second and so on). But the callback function successCallback of ag-grid, require an array of objects, that means that each row takes with it the keys related to column names (given r as a generic row, r["firstColumn"] is the element related to first column, r["secondColumn"] to the second and so on).

The second approach is totally infeasible for the server perspective, given the huge waste of memory used by the key-value mechanism. This leads to the need of a local conversion for each page received by the client:

client.appendCallback("message", function(message){
    message = JSON.parse(message.data);
    switch(message.command) {
        case "GetRows":
            if(message.res.code == 0) {
                var bulk = [];
                var arr = message.res.data;
                for (var i = 0, len = arr.length; i < len; i++) {
                    bulk[i] = {"par1" : arr[i][0], "par2" : arr[i][1], "par3" : arr[i][2], "par4" : arr[i][3], "par5" : arr[i][4], "par6" : arr[i][5]};
                }
                _data.successCallback(bulk);
            }
            break;
        default:
            break;
    }
},"ws");

What I need is a way to pass to successCallback the rows as array and not as objects avoiding the conversion part, like this:

client.appendCallback("message", function(message){
    message = JSON.parse(message.data);
    switch(message.command) {
        case "GetRows":
            if(message.res.code == 0) {
                _data.successCallback(message.res.data);
            }
            break;
        default:
            break;
    }
},"ws");

Any help will be appreciated

Answer 1

What about this ?

Fix the pageSize of something like 100.

Since you use server side paging you have implemented your own datasource : so when you're asked to load data : load [and convert] something like 10000 rows and store them in memory.

Then use your own intermediary paging : each time the grid ask for next rows, either get them from the memory or fetch the next 10k rows and [convert and] return only the 1st hundreth .

the [convert] part is your choice to place the conversion operation either when loading from the server either when asking the next 100 rows.

If the number of data is huge and you consider deploy this not only on your local computer, angular (or the browsers, i don't really know where it is) support gzip compressed data transparently.