Cloud Services to run Batch script when file is uploaded?

Question

I am looking to run a batch script on files that are uploaded from my website (one at a time), and return the resulting file produced by that batch script. The website is hosted on a shared linux environment, so I cannot run the batch file on the server.

It sounds like something I could accomplish with Amazon S3 and Amazon Lambda, but I was wondering if there were any other services out there that would allow me to accomplish the same task.

Answer 1

I would recommend that you look into S3 Events and Lambda.

Using S3 events, you can trigger a lambda function on puts and deletes in a S3 bucket and depending on your "batch file" task you may be able to achieve your goal purely in Lambda.

If you cannot use Lambda to replace the functionality of your batch file you can try the following:

• If you need to have the batch process run on a specific instance, take a look at Amazon SQS. You can have the S3 event triggered Lambda create a work item in SQS and your instance can regularly poll SQS for work to process.
• If you need something a bit more real time, you could use Amazon SNS for a push rather than pull approach to the above.
• If you don't need the file to be processed by a specific instance but you have to have a batch file run against it, perhaps you can have your S3 event triggered Lambda create an instance that has a UserData script that will sys prep the server as needed, download the s3 file, process the batch against it and then finally self terminate by looking up it's own instance ID via the EC2 Metadata service and calling the api method terminate instances.

Here is some related reading to assist with the above approaches:

Amazon SQS https://aws.amazon.com/documentation/sqs/

Amazon SNS https://aws.amazon.com/documentation/sns/

Amazon Lambda https://aws.amazon.com/documentation/lambda/

Amazon S3 Event Notifications http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

EC2 UserData http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-instance-metadata.html#instancedata-add-user-data

EC2 Metadata Service http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-retrieval

AWS Tools for Powershell Cmdlet Reference http://docs.aws.amazon.com/powershell/latest/reference/Index.html