Deploying code to Amazon EC2 instance

Question

I'm using Java to create EC2 instances from within Eclipse. Now I would like to push parts of the application to these instances so that these can process whatever needs processing and then send the results back to my machine.

What I'm trying to do is something along the lines of:

assignWork(){
    workPerformed = workQueue;
    workPerInstance = workQueue/numberOfInstances;
    while(workQueue > 0){
        netxInstance.doWork(workPerformed,workPerInstance);
        workPerformer -= workPerInstance;
    }
}

doWork(start, end){
    while(start>end){
       //process stuff
    start--;
    }
}

This way I could control exactly how many AMI's to instantiates depending on the volume of work at hand. I could instantiate them, send them specific code to process and then terminate them as soon as I receive the results.

Is this possible just using the AWS JDK?

Answer 1

It is, but consider that...

1. If you have SLAs, and they fall within SQS Limits (Maximum 4 Days), you could consider publishing your task queues into SNS/SQS, and use CloudWatch to track the number of needed instances.

2. If you have a clear division of roles (more like a workflow), and the long-running tasks are not of much concern and you can retry, also consider using AWS SWF instead. It goes a bit beyond of a SQS/SNS Combo, and I think it could fit nicely with CloudWatch (thats just a theory, I haven't looked further). Cons are the extreme assh*le AWS Flow Framework for writing the Workflow Processes

3. If your workload is predictable (say, around 5K processes to process today), meaning you have no need for real-time and you can batch those requests, then consider using Elastic MapReduce for this. Being Hadoop-based, it offers some such niceties, such as being able to resize your cluster on demand, and the obvious case of not having any vendor lock in at all.

4. Actually, if you want that manage and without many surprises, consider looking at options such as PiCloud and IronWorker . They were really made for situations just like the one you've just described.

If you have only a Queue and EC2, you can surely automate that. It only depends on how badly you want to coordinate these tasks, but I'm sure its possible.