堆栈内存溢出
  简体   繁体   English

Jenkins 并行管道构建

[英]Jenkins Parallel Pipeline Builds

 原文  2021-01-02 00:21:23  4   1    jenkins/ groovy/ build/ parallel-processing/ jenkins-pipeline

问题描述

Parallel pipelines seem to be reusing builds across the parallel branches.并行管道似乎正在跨并行分支重用构建。

I've been playing with parallel pipelines to understand how they work.我一直在玩并行管道来了解它们是如何工作的。 Assume 'FailRarely' is another job that 10% of the time fails, but 90% of the time it reports success.假设“FailRarely”是另一个有 10% 的时间失败但有 90% 的时间报告成功的作业。

My pipeline script is as follows:我的管道脚本如下：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely'
                    }
                }
            }
        }
    }
}

As you can see, it just attempts to run the job six times in parallel.如您所见，它只是尝试并行运行该作业六次。 Unfortunately, the log seems to indicate that it is using the same job for all of the parallel jobs:不幸的是，日志似乎表明它对所有并行作业使用相同的作业：

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
[Pipeline] build (Building FailRarely)
Scheduling project: FailRarely
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
Starting building: FailRarely #57
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Note that it used 'Job 57' for all of the pipeline builds.请注意，它对所有管道构建都使用了“Job 57”。

What am I doing wrong?我究竟做错了什么？

*** EDIT: ADD ADDITIONAL INFORMATION *** *** 编辑：添加其他信息 ***

Taking the same job, I embedded the tasks performed in FailRarely, so that calling another build from the pipeline was no longer being performed:做同样的工作，我嵌入了在 FailRarely 中执行的任务，因此不再执行从管道调用另一个构建：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 2') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 3') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 4') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 5') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
                stage ('Job 6') {
                    agent { label "linux" }
                    steps {
                       sh "python -c \"import random;import sys;sys.exit(random.randrange(10)==0)\""
                    }
                }
            }
        }
    }
}

This performs as I expected:这按我的预期执行：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\parallel-pipeline-2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] node
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@2
[Pipeline] // node
[Pipeline] // node
[Pipeline] {
[Pipeline] {
[Pipeline] }
[Pipeline] }
[Pipeline] }
Running on ubuntu-vbox in /home/jenkins/workspace/parallel-pipeline-2@3
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] sh
[Pipeline] sh
[Pipeline] sh
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
+ python -c import random;import sys;sys.exit(random.randrange(10)==0)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

So it certainly appears to be something with how the master job is calling the 'FailRarely' job.因此，这似乎与主作业如何调用“FailRarely”作业有关。

1 个解决方案

解决方案1
 0  2021-01-02 19:11:21

After trying a number of combinations across the main parallel job and the worker ('FailRarely') project, I kept coming back to a single conclusion:在主并行作业和工作者（'FailRarely'）项目中尝试了多种组合之后，我不断得出一个结论：

The main job only actually launches one worker if there are identical calls to the worker and reuses the result for all instances.如果对工作人员有相同的调用，则主作业实际上只会启动一个工作人员，并将结果重用于所有实例。

This 'feels' to me to be some kind of optimization in the the pipeline when launching builds, although I can find no documentation to that effect.对我来说，这“感觉”是在启动构建时管道中的某种优化，尽管我找不到关于这种效果的文档。

The workaround, however is simple.然而，解决方法很简单。 Once you add a parameter to the worker project, and provide a different value for the parameter, the main job realizes it needs to launch the job multiple times and does so correctly.一旦您将参数添加到工作项目，并为参数提供不同的值，主作业就会意识到它需要多次启动该作业并且这样做是正确的。

So, I added a string parameter to the FailRarely job (named IGNORED ) which isn't actually used.因此，我在FailRarely作业（名为IGNORED ）中添加了一个实际上并未使用的字符串参数。 Then, I added unique values for the parameter for each build command to make them unique.然后，我为每个build命令的参数添加了唯一值以使其唯一。

An example for the change to build within the main job:在主作业中build更改的示例：

build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]

The full modified main script:完整修改的主脚本：

pipeline {
    agent {
        label "master"
    }
    stages {
        stage ('Run jobs') {
            parallel {
                stage ('Job 1') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '1')]
                    }
                }
                stage ('Job 2') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '2')]
                    }
                }
                stage ('Job 3') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '3')]
                    }
                }
                stage ('Job 4') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '4')]
                    }
                }
                stage ('Job 5') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '5')]
                    }
                }
                stage ('Job 6') {
                    steps {
                        build job: 'FailRarely', parameters: [string(name: 'IGNORED', value: '6')]
                    }
                }
            }
        }
    }
}

And the output, showing that each instance is indeed created:而output，显示确实创建了每个实例：

Started by user Joe Marley
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\WINDOWS\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\workspace\Parallel Pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run jobs)
[Pipeline] parallel
[Pipeline] { (Branch: Job 1)
[Pipeline] { (Branch: Job 2)
[Pipeline] { (Branch: Job 3)
[Pipeline] { (Branch: Job 4)
[Pipeline] { (Branch: Job 5)
[Pipeline] { (Branch: Job 6)
[Pipeline] stage
[Pipeline] { (Job 1)
[Pipeline] stage
[Pipeline] { (Job 2)
[Pipeline] stage
[Pipeline] { (Job 3)
[Pipeline] stage
[Pipeline] { (Job 4)
[Pipeline] stage
[Pipeline] { (Job 5)
[Pipeline] stage
[Pipeline] { (Job 6)
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
[Pipeline] build
Scheduling project: FailRarely
Starting building: FailRarely #149Starting building: FailRarely #148
Starting building: FailRarely #147

Starting building: FailRarely #150
Starting building: FailRarely #151
Starting building: FailRarely #152
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Problem solved via workaround.通过变通方法解决了问题。 I'll leave this open in case someone can give me a better answer for a few days/weeks.我会保持开放状态，以防有人可以在几天/几周内给我一个更好的答案。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Jenkins声明式管道并行构建 - Jenkins declarative pipeline parallel builds 通过Jenkins 2.0 Pipeline中的并行执行访问构建 - Access builds from Parallel execution in Jenkins 2.0 Pipeline Jenkins声明式管道为并行构建创建自定义工作区 - Jenkins declarative pipeline create custom workspace for parallel builds Jenkins Pipeline - 如何从并行构建中获取日志 - Jenkins Pipeline - how to get logs from parallel builds 詹金斯与并行构建 - Jenkins with Parallel Builds 詹金斯管道（并行&&动态）？ - Jenkins pipeline (parallel && dynamically)? Jenkins管道并行不执行 - Jenkins pipeline parallel not exeucting Jenkins 管道条件分支构建 - Jenkins pipeline conditional branch builds Jenkins 流水线并行 - Jenkins pipeline with parallel Jenkins 并行声明式管道 - Jenkins parallel declarative pipeline
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM