How to run EMR steps first sequentially and then in parallel?

Question

This process can be achieved easily with the help of orchestration engine like Step Function but for some reason we cannot use. Here, I want to understand can we do some kind of code manipulation to run parallel execution after step2 where step 3 and step4 will run in parallel and after step 4 it will run step 5 in sequence.

Any reference how can I achieve this or anyone does so far? Thanks

Answer 1

As you said you can't go with Step Function for some reason, you can try Oozie.

Oozie is a workflow scheduler system to manage Apache Hadoop jobs. Its also a EMR managed service, one click to install. Oozie jobs are Directed Acyclical Graphs (DAGs).