oozie：并行运行数百个作业

Question

Initially we have five tables to process, so we created fork for 5 tables as below. but now we need to process 125 tables in parallel. If I fork all 125 tables, workflow.xml grow bigger and unable to maintain. How can I configure workflow to process all the 125 tables in parallel.

<start to="fork-966"/>
<fork name="fork-966">
    <path start="table1_sqoop" />
    <path start="table2_sqoop" />
    <path start="table3_sqoop" />
    <path start="table4_sqoop" />
    <path start="table5_sqoop" />
</fork>

Help appreciated.

Answer 1

Sound like 125 may not be the limit ....

Also, if you need to read from 125 tables I suggest re-think your design.

For your current problem , you can do 1 of the next:

1. Fork 125 times
2. Include sub workflow. This can be param'
3. Create bundle that run 125 workflows, so you will write you wf' only once and bundle can re-run it 125 times. Also if one is fail the rest are still running

Again - I think that you need to re-think you design