在向AWS Data Pipeline添加EC2资源时，如何指定EBS卷？

Question

When I try to create an EC2 Resource with a AWS Data Pipeline, I don't see and option for defining EBS volume that will be associated with that compute engine. Is it possible to set the volume size? If yes, can someone give me an example.

Answer 1

No, currently you cannot specify EBS volumes in Data-Pipelines Ec2Resource object.

Often times ROOT volume is used as staging directory for most of the Data-Pipeline activities which is currently limited to 8GB for Data-Pipeline provided default AMI's.

So, you can make your own AMI out of an Ec2 instance with increased EBS root volume and include that AMI in the Resource object(Image-id field) of Data-Pipelines.

Tip: You can check the AMI-id of Data-pipelines launched ec2 instances in Ec2 console.Use that AMI to create an ec2 instance with increased EBS ROOT volume and Make Image(AMI) from this instance with even more volume size. This way you dont need to choose from a list of AMI's and you will preserve visualization type that is required for launching particular instance types.

Answer 2

Currently, datapipeline service does not support an option to specify ebs volumes to attach to a ec2 resource. But, you can write a shell command activity ( http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html ) which can create ebs volume and attach it to the instance.

You can then run the shell command activity on the ec2 resource.

You can use aws cli for creating and attaching ebs volumes assuming that you have appropriate role on the launched ec2 instance.

you can also make use of ec2 metadata service: http://169.254.169.254/2009-04-04/meta-data/ to get instance info like availability zone etc... so that you can create ebs volume in the same availability zone

related post: https://serverfault.com/questions/126408/creating-ebs-volumn-from-and-attaching-to-the-same-instance