如何从服务器向客户端发送大量数据集

Question

I have a Requirement to send Data from Server (Tomcat : Java Process, Odata API's) to Client (React Based)

Data can range from few KB's to Hundred's of MB (Say 700 MB) which is retrieved from DB : RedShift , Processed and Sent to Client.

There can be multiple clients accessing at the same time as well to keep more stress on the system.

We added Pagination so that data for that page alone is loaded, but we have a functionality to export complete data set in CSV format.

Processing of all the data is consuming lot of memory and application's heap gets exhausted sometime, Increasing heap is not the solution expected, I want to know from Application side anything can be done to Optimize system resources.

Kindly suggest what could be the best way to transfer data also whould like to see if there are any other kind of API(Streaming) which can help me here

Answer 1

Can you change the integration between client and your system?

Something like: the client sends the request to export a CSV with a callback url in payload. You put this request in a queue (rabbitmq). The queue consumer process the request, generate the CSV and put it in a temporary area (S3 or behind a NGINX). Then your consumer notifies the client in the callback url with then new url for the client to download de full CSV.

This way, the system that process the incoming requests don't use too much heap. You only need to scale the queue consumer, but it's more easy because the concurrency is your configuration of how many consumers are consuming the messages, not the incoming requests from the clients.