Running non-matrix stages in parallel to a matrix in Jenkins declarative-pipeline

Question

I'm converting our enterprise build so that it uses the 'matrix' directive ( https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/ ), but am having trouble optimizing it. In the pre-matrix world, we have a build step (produces jars), followed by a parallel 'package' step (produces linux and windows distros), essentially followed by a 'system test' phase (execute the windows and linux distros on various JVMs), and we do some 'code quality' inspection in parallel to the package-and-system-test phases.

Much of this seems to lend itself to using 'matrix'. So, the 'packaging' is clearly a matrix to get windows and linux platforms crafted. And the 'system test' is another dual-axis matrix of platform and jvm. I can easily make one matrix follow the other. So far, so good.

However, I'm stuck with the 'code quality' stuff as an outlier. Is there a way to run those stages parallel to the matrix runs. They are independent once the build is created (they do not need to wait for packaging). They are also time consuming, so running them serially with the two matrix phases makes the build longer. Jenkins complains if you put a matrix inside a parallel stage.

Any ideas on how to run non-matrix stages in parallel to a matrix?

Answer 1

I have a work-around to my own question.

According to the documentation ( https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-matrix ) what I am looking for is not possible. "Each cell in a matrix can include one or more stages to be run sequentially using the configuration for that cell. Note that a stage must have one and only one of steps, stages, parallel, or matrix. It is not possible to nest a parallel or matrix block within a stage directive if that stage directive is nested within a parallel or matrix block itself."

But.... you can cheat. I was able to turn the matrix into a generic dispatch queue. ie each combination invokes N stages - for me, 2 and I call them "prepare" and "execute". I pass an additional "matrixtype" argument in. The matrixtype gets 1 value for the matrix itself, as well as additional values for each line of the non-matrix. I then use the matrix 'excludes' to insure that the non-matrix lines execute only once. Effectively, this folds the non matrix into the matrix differentiated by matrix type.

Original (parallel-stages in series with a subsequent matrix)
    stage {
        parallel {
           stage("alpha") {
                alpha(..)
           }
           stage("beta") {
               beta(..)
           }
           // etc
       }
    }
    stage {
        matrix {
            axes {
                axis {
                    name 'ORIGAXIS'
                    values 'FOO','BAR','BAZ'
                }
            }
            stages {
               stage("First") {
                  originalFirst( ...)
               }
               stage("Second") {
                  originalSecond(...)
               }
            }
         }
      }

Replacement (parallel folded into matrix)
    stage {
        matrix {
            axes {
                axis {
                    name 'MATRIXTYPE
                    values 'ORIGINAL', 'ALPHA', 'BETA'
                axis {
                    name 'ORIGAXIS'
                    values 'FOO','BAR','BAZ'
                }
                excludes {
                    // Execute Alpha and Beta only once (during 'FOO')
                    exclude {
                        axis {
                            name 'MATRIXTYPE'
                            values 'ALPHA', 'BETA'
                        }
                        axis {
                            name 'ORIGAXIS'
                            values 'BAR','BAZ'
                        }
                    }
                }
            }
            stages {
               stage("First") {
                  dispatchFirst( "${MATRIXTYPE}", ...)
               }
               stage("Second") {
                  dispatchSecond( "${MATRIXTYPE}", ...)
               }
            }
         }
      }

The dispatchFirst(..) and dispatchSecond(..) are then simple dispatch methods in the shared lib that examine matrixtype, and invoke originalFirst(..), originalSecond(..), alpha(..), beta(..) or a no-op as appropriate. It's slightly clumsy, and amounts to shoehorning the parallel stages into the matrix, but it works. And, you get the benefit of parallelization (build speed-optimization)

Hopefully in the future, there will be something more elegant.

Answer 2

I came up with a somewhat IMHO "better" solution:

stage {
    matrix {
        axes {
            axis {
                name 'ORIGAXIS'
                values 'ALPHA','BETA','BAR','BAZ'
            }
        }
        stages {
           stage ("alpha") {
               when { expression { env.ORIGAXIS == "ALPHA" } }
               steps {
                   alpha()
               }           
           stage ("beta") {
               when { expression { env.ORIGAXIS == "BETA" } }
               steps {
                   beta()
               }
           }
           stage ("Tests") {
               when { allOf
                   expression { env.ORIGAXIS != "ALPHA" }
                   expression { env.ORIGAXIS != "BETA" }
               }
               stages {
                   stage("First") {
                      originalFirst( ...)
                    }
                   stage("Second") {
                      originalSecond(...)
                   }
                }
            }
     }
}

Of course, the final layout is not perfect. But, it works, is not that much cumbersome and still easily maintainable.