在AWS中具有t2.micro实例的集群

Question

I am new to Amazon Web Services (AWS) & I am using the free tier t2.micro right now ( 1 CPU and 1 GB memory). Doing some backtesting/ simulation stuff and it seems free tier is quite inadequate. Pretty slow actually. Thus thinking of options which will help me to run my code at a faster speed for few hours.

Option 1 : I can 1 buy CPU optimized/ higher Memory instance ( 4 cores and 4 GB RAM for example ) and then make an image of my t2.micro and run my stuff in that new one. It will be expensive though if I keep it running, so I need to "stop" the instance when I am not working ( or nothing is running ) to reduce the cost.

Option 2 : I can buy spot instances. I am not sure how to use the CPU and RAM of that spot instance from my existing t2.micro. Can I create a temporary grid/cluster where my Head Node will be running in my t2.micro but compute node will be the spot instance ( higher CPU and RAM ), thus all my calculations, etc will be using the spot instance.

My question : Is the Option 2 possible ? I program everything in python and I have all the relevant softwares/python IDEs etc are already installed in my t2.micro instance.

Any existing grid/cluster software I can use right now ? I don't know C++, Csharp, Java etc. Know only phython & R so any programming stuff I need to do to build a grid/cluster must use Python :)

Thank you in advance.

Answer 1

Much of what you are asking depends upon your use-case. For example, if you have work continually arriving then you will need capacity continually available. However, if it is more batch-oriented then you could start/stop capacity and even use the new Amazon Batch service that can allocate resources when needed and remove them when jobs are finished.

Some things to note:

• You can change the Instance Type when an instance is stopped. So, your t2.micro instance can be changed to a large instance (eg m4.xlarge) by stopping it, changing the instance type and starting it again.
• The t2.micro Instance Type is actually extremely powerful when CPU burst credits are available, but it limited in capability when all CPU Credits are consumed. A good machine for bursty workloads, but not continual workloads.
• Spot instances are great, but please note that they will be terminated if the Spot Price goes above your bid price. Prices vary by region, Availability Zone within a region and instance type, so you could launch instances with a variety of attributes (different AZs, different instance types) that would make it unlikely that you would lose all capacity at the same time. Spot can save considerable costs and is well worth investigating.
• Take a look at Amazon EMR , which runs Hadoop to provide parallel processing across a cluster of instances.
• Stop instances when you don't need them. That's the best way to get good value!