Recommendations for Hadoop on EC2?

Question

When running Hadoop in EC2, I seem to have two options:

• A: Manage the cluster myself, using the EC2-specific shell scripts that come with Hadoop.
• B: Use Elastic MapReduce, and pay a little extra for the convenience.

I'm leaning towards B, but I'd appreciate some advice from people with more experience. Here are my questions:

1. Are there any tasks that can be done with one of these methods but not the other?
2. Are there other options besides these two that I'm overlooking?
3. If I choose B, how easy would it be to go back to A? That is, what's the danger of vendor lock-in?

Answer 1

第三种选择：您可以使用apache whirr在ec2上设置hadoop集群（也支持机架空间）

Answer 2

I have been told by people close to the Amazon Elastic MapReduce (EMR) development team that there are at least two other advantages to using EMR: a) Amazon is actively applying bug fixes and performance enhancements to the Hadoop code base used on EMR, and b) Amazon employs a high performance network between EMR servers and S3 servers that may not be available between EC2 servers and S3 servers.

UPDATE: See @mat's comments that refute the rumored advantages of using EMR.

Answer 3

Disclaimer: I'm the founder of Axemblr.com

There are also commercial alternatives you can use. Axemblr Tool for Cloudera CDH3 is a tool we are building that can deploy a cluster in just a few minutes with all you need (including Cloudera Hue, Mahout & Pig).

We are also building an alternative to EMR that's fully compatible from an API perspective, targeted at private clouds.

If you are wondering why it makes sense to run CDH on EC2 rather than EMR see:

http://www.quora.com/What-are-the-advantages-disadvantages-running-Clouderas-distribution-for-Hadoop-on-EC2-instances-rather-than-using-Amazons-Elastic-Map-Reduce-Service