Do I need to use queue for uploading file to Amazon S3 from my Node.js Server?

Question

I have this question in my mind for a long time. Does file upload to Node.js Server won't block it's thread ? I here don't mean getting the response for single upload.

For example, consider 100 users uploading pictures(or large files) to the server. Once the file is uploaded to my server, I will move it Amazon S3. Now,

1. How Node.js handles this 100 requests ? Separate callback for each 100 requests in parallel ?
2. I don't want users to wait until it is uploaded to S3, once file is uploaded to Server, it should be transferred to S3 separately and response should be sent immediately.
3. Will responding back to user, before the file transfer to S3 completes, kill that(the upload to S3) process ?
4. Should I use queue to process this transfer to S3 ? or not needed ?

Note: Don't mark it as duplicate unless you found a SO question with answer to all my questions.

Answer 1

Since you mentioned that you are not processing the files on your server, you should directly upload the files to S3 without making it go through your server. This would be the most optimal work flow. That is, user --> S3 instead of user --> node--> S3 .

Now to your specific question, a file upload itself will not block Node.js.

All the I/O operations is handled by Node.js is using multiple threads internally; it's the programming interface to that I/O functionality that's single threaded, event-based, and asynchronous.

More at: Will Node.js get blocked when processing large file uploads?

So, the answer to your questions are:

1. Node.js will initiate file upload for those 100 requests one by one and spawn threads for each and when each one is done, the callback will be put into the event loop queue.

2. If you don't want users to wait (also, waiting will possibly timeout anyway) and if you want to transfer to S3 separately, use a queue.

3. Depends on your implementation but with a queue it should not.

4. If you go user -->node-->s3 , yes. Otherwise, chances are your users will timeout long before upload to S3 is complete.

You should probably follow this queue route if you are doing any sort of processing with your files. Otherwise, use my original recommendation. If you need help with implementing that, there's plenty of examples available.