Node JS - file generator architecture

Question

Need to add file generator REST API endpoint to web app. So far I've came up with following idea:

1. client sends file parameters to endpoint
2. server receives request and using AMQP sends parameters to dedicated service
3. dedicated service creates a file, puts it into server folder and sends responce that file created with file name
4. endpoint sends response to client with file

I'm not sure if's a good idea to keep REST request on a server for so long. But still don't want to use email with generated link or sockets.

Do I need to set timeout time in request so it will not be declined after a long wait time?

As far as I know maximum timeout is 120sec for rest api call. If it takes more time for the service to create a file then I need to use sockets, is that right?

Answer 1

The way I've handled similar is to do something like this:

1. Client sends request for file.
2. Server adds this to a queue with a 'requested' state, and responds (to the client) almost immediately with a reponse which includes a URL to retrieve the file.
3. Some background thread/worker/webJob/etc is running in a separate process from the actual web server and is constantly monitoring the queue - when it sees a new entry appear it updates the queue to a 'being generated' state & begins generating the file. When it finishes it updates the queue to a 'ready' state and moves on...
4. when the server receives a request to download the file (via the URL it gave the client), it can check the status of the file on the queue. If not ready, it can give a response indicating this. If it IS ready, it can simply respond with the file contents.
5. The Client can use the response to the initial request to re-query the url it was given after a suitable length of time, or repeatedly query it every couple of seconds or whatever is most suitable.

You need some way to store the queue that is accessible easily by both parts of the system - a database is the obvious one, but there are other things you could use...

This approach avoids either doing too much on a request thread or having the client 'hanging' on a request whilst the server is compiling the file.

That's what I've done (successfully) in these sorts of situations. It also makes it easy to add things like lifetimes to the queue, so a file can automatically 'expire' after a while too...