在多个 AWS 实例上运行并行 Python 代码

Question

I have a Python algorithm that can be parallelized fairly easily.

I don't have the resources locally to run the whole thing in an acceptable time frame.

For each work unit, I would like to be able to:

1. Launch an AWS instance (EC2?)
2. Send input data to the instance
3. Run the Python code with the data as input
4. Return the result and aggregate it when all instances are done

What is the best way to do this?

Is AWS Lambda used for this purpose? Can this be done only with Boto3?

I am completely lost here.

Thank you

Answer 1

A common architecture for running tasks in parallel is:

• Put inputs into an Amazon SQS queue
• Run workers on multiple Amazon EC2 instances that:
  • Retrieve a message from the SQS queue
  • Process the data
  • Write results to Amazon S3
  • Delete the message from the SQS queue (to signify that the job is complete)

You can then retrieve all the results from Amazon S3. Depending on their format, you could even use Amazon Athena to run SQL queries against all the output files simultaneously.

You could even run multiple workers on the same instance if each worker is single-threaded and there is spare CPU available.