Run independent `PythonScriptStep` steps in parallel

Question

In my pipeline multiple steps are independent and so I would like them to run in parallel based on input dependencies.

As the compute I use has multiple nodes I would have expected this to be the default.

For example:

All 3 upper steps should run in parallel, then both f.netune steps in parallel as soon as their inputs are satisfied and the same for rgb_test .

Currently only 1 step runs at a time, the other are Queued .

Answer 1

It ended up being because of vCPU quota.

After increasing the quota, parallel tasks can run at the same time as expected.

Answer 2

I agree this is weird! Are you sure your compute target's max_nodes is at least 3?