Nodejs API, Docker (Swarm), scalability and storage

Question

I programmed an API with nodejs and express like million others out there and it will go live in a few weeks. The API currently runs on a single docker host with one volume for persistent data containing images uploaded by users.

I'm now thinking about scalability and a high availability setup where the question about network volumes come in. I've read a lot about NFS volumes and potentially the S3 Driver for a docker swarm.

From the information I gathered, I sorted out two possible solutions for the swarm setup:

Docker Volume Driver

• I could connect each docker host either to an S3 Bucket or EFS Storage with the compose file
• Connection should work even if I move VPS Provider
• Better security if I put a NFS storage on the private part of the network (no S3 or EFS)

API Multer S3 Middleware

• No attached volume required since the S3 connection is done from within the container
• Easier swarm and docker management
• Things have to be re-programmed and a few files needs to be migrated
• On a GET request, the files will be provided by AWS directly instead of the API

Please, tell me your opposition on this. Am I getting this right or do I miss something? Which route should I take? Is there something to consider with latency or permissions when mounting from different hosts? Tipps on S3, EFS are definitely welcome, since I have no knowledge yet.

Answer 1

I would not recommend saving to disk, instead use S3 API directly - create buckets and write in your app code.

If you're thinking of mounting a single S3 bucket as your drive there are severe limitations with that. The 5Gb limit. Anytime you modify contents in any way the driver will reupload the entire bucket. If there's any contention it'll have to retry. Years ago when I tried this the fuse drivers weren't stable enough to use as part of a production system, they'd crash and you have to remount. It was a nice idea but could only be used as an ad hoc kind of thing on the command line.

As far as NFS for the love of god don't do this to yourself you're taking on responsibility for this on yourself.

EFS can't really comment, by the time it was available most people just learned to use S3 and it is cheaper.