Limit concurrent executions on AWS Data Pipeline

Question

Is there a way to limit concurrent execution on an AWS Data Pipeline? We need to limit simultaneous executions to 1.

Something similar to what Oozie has with the <concurrency> property?

From the oozie docs:

concurrency: The maximum number of actions for this job that can be running at the same time. This value allows to materialize and submit multiple instances of the coordinator app, and allows operations to catchup on delayed processing. The default value is 1 .

Answer 1

You can use maxActiveInstances field under EC2Resource / EmrCluster to achieve this.

References -

https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-emrcluster.html

https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-ec2resource.html