How can I transfer data while generating it using REST API?

Question

How can I transfer data while generating using REST API without storing it in any temporary memory storage?

Basically I have some data stored in database which ideally I needed to write to a file, store it in server and transfer to client. But server have space restriction and not allowing me to store so large file. What should I do in that case?

Answer 1

Don't know if this best approach for your scenario, If anyone has a better solution than this please provide. The constraint is that you have limited size on your server. following is my approach to solve your problem

1. Currently, you have one REST API (FilePart API) which is deployed on one server which has limited storage. (let's call it server 1)
2. Make this FilePart REST API deployed on server 1 to create file parts (instead of the whole file) based on pagination data and return generated part file.
3. Deploy one more application (FileCombine API) on a different server (let's call it server 2) which will call FilePart REST API on the server 1 and sends pageNo and pageSize as parameters. This call will accept generated filePart as response and stored on server 2.
4. FileCombine API will make multiple calls to FilePart API until all parts get downloaded.
5. Once all parts get downloaded on server 2, FileCombine API will combine all the fileParts and send it back to the client.

So your architecture will be like.

Client --calls--> FileCombineAPI(Server 2) --mutliple calls--> FilePartAPI(Server 1)

Server 1: Limited space.

Server 2: Enough space to handle operation.