AWS Data Pipeline Service创建新的ec2实例

Question

I have created a new DataPipeline to stop some instances eg tagged as auto-stop/auto-start .

My command is sth like this:

aws ec2 describe-instances --region us-west-2 --filter "Name=tag:auto-stop,Values=yes" "Name=instance-state-name,Values=running" --query 'Reservations[*].Instances[*].[InstanceId]' --output text |xargs aws ec2 stop-instances --region us-west-2 --instance-ids

ie Stop all instances which are running and tagged as auto-stop:yes in given region

Now as soon as scheduler starts an activity, I see a new t1.micro instance is started with a public ip assigned. My question is,is this behavior normal for Data Pipelines ?

Please provide me any link to documentation and how much I ll be charged for this activity.

If the creation of the associated instance is a normal thing,what is the life-cycle of that instance ?

Answer 1

Yes, this behavior is normal. The t1.micro instance is the one executing the command.

An Ec2Resource object is set up in your Data Pipeline. It can be customized, see documentation: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-ec2resource.html

By default it's a t1.micro instance that terminates after 50 minutes. You are charged for this instance (EC2 pricing: https://aws.amazon.com/ec2/pricing/ ).

As mentioned in one of EC2 knowledge base article https://aws.amazon.com/premiumsupport/knowledge-center/stop-start-ec2-instances/ :

An Amazon EC2 t1.micro instance is started as the host environment for execution of a data pipeline. EC2 instances started for this purpose run for a default timeout period of 50 minutes. All resources used to host execution of a data pipeline are accrued to your account. Executing pipelines to stop and restart an EC2 instance for 100 minutes or less will use the same amount of resources as would be used by simply letting an EC2 t1.micro instance continue to run. You should stop and restart one or more EC2 t1.micro or larger instances for more than 100 minutes (50 minutes to stop and 50 minutes to start) to ensure that the method described in this article does not consume more resources than are conserved.

Answer 2

The life cycle of the Ec2 Resource is tied to the activities that need to run on that resource. Resource is terminated when activities are complete or the 'terminateAfter' timeout is reached. I believe, as it stands today, AWS Ec2 does not charge for t1.micro instances.