Amazon EC2 scaling and upload temporary folder

Question

I have an application based on php in one amazon instance for uploading and transcoding audio files. This application first uploads the file and after that transcodes that and finally put it in one s3 bucket. At the moment application shows the progress of file uploading and transcoding based on repeatedly ajax requests by monitoring file size in a temporary folder.

I was wondering all the time if tomorrow users rush to my service and I need to scale my service with any possible way in AWS.

A: What will happen for my upload and transcoding technique?

B: If I add more instances does it mean I have different files on different temporary conversion folders in different physical places?

C: If I want to get the file size by ajax from http://www.example.com/filesize up to the finishing process do I need to have the real address of each ec2 instance (i mean ip,dns) or all of the instances folders (or folder)?

D: When we scale what will happen for temporary folder is it correct that all of instances except their lamp stack locate to one root folder of main instance?

I have some basic information about scaling in the other hosting techniques but in amazon these questions are in my mind.

Thanks for advice.

Answer 1

It is difficult to answer your questions without knowing considerably more about your application architecture, but given that you're using temporary files, here's a guess:

1. Your ability to scale depends entirely on your architecture, and of course having a wallet deep enough to pay.

2. Yes. If you're generating temporary files on individual machines, they won't be stored in a shared place the way you currently describe it.

3. Yes. You need some way to know where the files are stored. You might be able to get around this with an ELB stickiness policy (ie traffic through the ELB gets routed to the same instances), but they are kind of a pain and won't necessarily solve your problem.

4. Not quite sure what the question is here.

As it sounds like you're in the early days of your application, give this tutorial and this tutorial a peek. The first one describes a thumbnailing service built on Amazon SQS, the second a video processing one. They'll help you design with best AWS practices in mind, and help you avoid many of the issues you're worried about now.

Answer 2

One way you could get around scaling and session stickiness is to have the transcoding update a database with the current progress. Any user returning checks the database to see the progress of their upload. No need to keep track of where the transcoding is taking place since the progress gets stored in a single place.

However, like Christopher said, we don't really know anything about you're application, any advice we give is really looking from the outside in and we don't have a good idea about what would be the easiest thing for you to do. This seems like a pretty simple solution but I could be missing something because I don't know anything about your application or architecture.