Running Jenkins steps in parallel based on number of cores with declarative pipeline

Question

I am using Jenkins with a declarative pipeline. I have a Jenkins machine with 8 cores running any number of steps (say, 20) in parallel. The problem is that too many tasks run in parallel and some tests fail due to timeout issues.

I need a way to tell Jenkins to run 20 tests in parallel but only 8 at a time. This way, I will give it the longer running steps first followed by faster steps. Once it finishes one of the longer steps, it should immediately start the next while the other 7 are still running. Something like this but only 8 at a time:

            steps {
                parallel(
                    runtest1:
                    {
                        dostep()
                    },
                    runtest2:
                    {
                        dostep()
                    },
                    runtest3: 
                    {
                        dostep()
                    },
                    runtest4: 
                    {
                        dostep()
                    },
                    runtest5:
                    {
                        dostep()
                    },
                    runtest6:
                    {
                        dostep()
                    },
                    runtest7:
                    {
                        dostep()
                    },
                    runtest8:
                    {
                        dostep()
                    },
                    runtest9:
                    {
                        dostep()
                    }
                    runtest10:
                    {
                        dostep()
                    },
                    runtest11:
                    {
                        dostep()
                    },
                    runtest12:
                    {
                        dostep()
                    },
                    runtest13: 
                    {
                        dostep()
                    },
                    runtest14: 
                    {
                        dostep()
                    },
                    runtest15:
                    {
                        dostep()
                    },
                    runtest16:
                    {
                        dostep()
                    },
                    runtest17:
                    {
                        dostep()
                    },
                    runtest18:
                    {
                        dostep()
                    },
                    runtest19:
                    {
                        dostep()
                    },
                    runtest20:
                    {
                        dostep()
                    }
                )

            }
        }

Today with parallel() it will only run all tests then when it is done with everything, it will go to the next stage where it will run the next parallel steps.

Answer 1

Your code will try to run all these tests in parallel on the same agent. That might not be what you want. You can try running entire stages in parallel on different agents: https://jenkins.io/blog/2017/09/25/declarative-1/

stages {
    stage('Run Tests') {
        parallel {
            stage('Test 1') {
                agent { label "8-core-agent" }
                steps {
                    doTheTest()
                }
            }
            stage('Test 2') {
                agent { label "8-core-agent" }
                steps {
                    doTheTest()
                }
            }
        }
...

If you're just running your build on the master or a single agent, you could probably use the method above to divide your tests into stages: stage('test 1-8') ... stage('test 9-16') etc

If you set the number of executors on 8-core-agent to 8, Jenkins should schedule 8 of these tasks at a time on it, until they are all done.