Jenkins Parallel Pipeline Builds

Question

Parallel pipelines seem to be reusing builds across the parallel branches.

I've been playing with parallel pipelines to understand how they work. Assume 'FailRarely' is another job that 10% of the time fails, but 90% of the time it reports success.

My pipeline script is as follows:

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
            }
        }
    }
}

As you can see, it just attempts to run the job six times in parallel. Unfortunately, the log seems to indicate that it is using the same job for all of the parallel jobs:

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Note that it used 'Job 57' for all of the pipeline builds.

What am I doing wrong?

*** EDIT: ADD ADDITIONAL INFORMATION ***

Taking the same job, I embedded the tasks performed in FailRarely, so that calling another build from the pipeline was no longer being performed:

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 2') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 3') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 4') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 5') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 6') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
            }
        }
    }
}

This performs as I expected:

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\parallel-pipeline-2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] // node
[Pipeline] // node
[Pipeline] {
[Pipeline] {
[Pipeline] }
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

So it certainly appears to be something with how the master job is calling the 'FailRarely' job.

Answer 1

After trying a number of combinations across the main parallel job and the worker ('FailRarely') project, I kept coming back to a single conclusion:

The main job only actually launches one worker if there are identical calls to the worker and reuses the result for all instances.

This 'feels' to me to be some kind of optimization in the the pipeline when launching builds, although I can find no documentation to that effect.

The workaround, however is simple. Once you add a parameter to the worker project, and provide a different value for the parameter, the main job realizes it needs to launch the job multiple times and does so correctly.

So, I added a string parameter to the FailRarely job (named IGNORED ) which isn't actually used. Then, I added unique values for the parameter for each build command to make them unique.

An example for the change to build within the main job:

build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]

The full modified main script:

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '1')]
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '2')]
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '3')]
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '5')]
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '6')]
                    }
                }
            }
        }
    }
}

And the output, showing that each instance is indeed created:

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
Starting building: FailRarely #149Starting building: FailRarely #148
Starting building: FailRarely #147

Starting building: FailRarely #150
Starting building: FailRarely #151
Starting building: FailRarely #152
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Problem solved via workaround. I'll leave this open in case someone can give me a better answer for a few days/weeks.