在同一管道上迭代云数据融合

Question

Usecase to execute-

1. Excel with multiple tabs uploaded to cloud storage
2. Cloud function trigger calls cloud datafusion pipeline
3. Pipeline reads the file uses wrangler to read the individual sheets and write to separate tables as per the sheet

Though I am using params to define the sheetname in wrangler(as in usage for usable pipelines), I am unable to figure out how to either iterate to run for each tab, or use cloud function to span 20 calls to same pipeline with parameter as each tab name. I do not think running parallel paths in same pipeline is going to help or is a good solution. Any help appreciated.

Answer 1

To run parallel pipelines, we can iterate through API call using cloud functions. Though I am not sure how many instances are allowed in a single call to CDF, I could run three parallel runs of same pipeline.