Can I run 1000 AWS micro instances simultaneously?

Question

I have a simple pure C program, that takes an integer as input, runs for a while (let's say an hour) and then returns me a text file. I want to run this program 1000 times with input integers from 1 to 1000.

Currently I'm running this program in parallel (4 processors), what takes 250 hours. The program is such that it fits in a AWS micro instance (I've tested it). Would it be possible to use 1000 micro instances in AWS to do the whole job in one hour? (at a cost of ~20$ - $0.02/instance)?

If it is possible, does anybody have some guidlines on how to do that? If it is not, does anybody have an also low-budget alternative to that?

Thanks a lot!

Answer 1

In order to achieve this, you will need to:

• Create a S3 bucket to store your bootstrapping scripts, application, input data and output data

• Custom Lightweight AMI: you might want to create a custom lightweight AMI which knows about downloading the bootstrapping script

• Bootstrapping Script: will download your software from your S3 bucket, parse a custom instance tag which will contain the integer [1..1000] and download any additional data.

• Your application: does the processing stuff.

• End of processing script is which uploads the result to a another result S3 bucket and terminates the instance , you might also want to send a SNS notification to communicate the end of processing status.

• If you need result consolidation you might want to create another instance and use it as a coordinator, waiting for all "end of processing" notifications in order to finish the processing. In this case you might consider using in the future the Amazon's Hadoop map reduce engine, since it will do almost all this heavy lifting for free.

Answer 2

You didn't specify what language you'd like to do this in and since the app you use to deploy your program doesn't have to be in the same language I would recommend C#. There are a number of examples around the web on how to programmatically spawn new Amazon Instances using the SDK. For example:

How to start an Amazon EC2 instance programmatically in .NET

You can create your own AMIs with the program already present but that might end up being a pain if you want to make adjustments to it since it will require you to recreate the entire AMI. I'd recommend creating an extra instance or simply hosting the program in a location which is accessible from a public URL. Then I would create some kind of service which would be installed on the AMI ahead of time to allow me to specify the URL to download the app from along with whatever command line parameters I wanted for that particular instance.

Hope this helps.

Answer 3

Be carefull by default you can only spawn 20 instance by zone. You need to ask amazon in order to use more instances.

If you want low cost but don't care about delay you should use spot instances.