Amazon SQS, Boto, and Python: Fetching and Running a Script on Amazon EC2

Question

I am trying to run a set of python scripts on Amazon EC2. This help guide suggests using Amazon SQS to manage a queue. From this guide, I understand how to boot up an EC2 instance and send a job to the queue from my computer, but I do not understand how to set up the EC2 instance such that it will run the python script after it is fetched from the queue and then save the result to Amazon S3. How do I fetch, run on the EC2 server, and save the results to S3 of a python script?

Answer 1

In the mentioned guide you can find a link to a github repository with sample code. This is a working example, answering your question. It is well documented and easy to read.

Namely, look closer on pdf/tasks.py source code. In short, it use ec2.run_instances to startup ec2 instance with a bootstrap bash script that:

• updates instance with apt-get
• in a loop uses boto.connect_sqs to connect to SQS and fetch a task
• performs task
• uses boto.connect_s3() to connect to S3 and upload results
• shutdowns instance when script finishes.

UPDATE

Moreover, CheckResponseQueueTask is a good example of pereodic dequeuing messages with Celery.PeriodicTask from SQS and processing them. Look BOOTSTRAP_SCRIPT , source lines 51-57, for sample code illustrating results uploading to S3. And CheckQueueLevelsTask illustrates new instance booting and bootstraping with queue processor, in case of amount of tasks exceeding number of instances. Lines 214-244 for reference.

Answer 2

One way to so this would be to run Celery on your EC2 instance which is able to use AWS SQS as a task broker out of the box.

Information on using Celery and AWS SQS can be found here: http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html

Celery also supports concurrency and multiple workers. This means you can have multiple EC2 instances running Celery listening to the one AWS SQS queue and the tasks will distribute themselves among your instances automatically.