Can I create dynamically stages in a Jenkins pipeline?

Question

I need to launch a dynamic set of tests in a declarative pipeline. For better visualization purposes, I\'d like to create a stage for each test. Is there a way to do so?

Answer 1

You might want to take a look at this example - you can have a function return a closure which should be able to have a stage in it.

This code shows the concept, but doesn't have a stage in it.

def transformDeployBuildStep(OS) {
    return {
        node ('master') { 
        wrap([$class: 'TimestamperBuildWrapper']) {
...
        } } // ts / node
    } // closure
} // transformDeployBuildStep

stage("Yum Deploy") {
  stepsForParallel = [:]
  for (int i = 0; i < TargetOSs.size(); i++) {
      def s = TargetOSs.get(i)
      def stepName = "CentOS ${s} Deployment"
      stepsForParallel[stepName] = transformDeployBuildStep(s)
  }
  stepsForParallel['failFast'] = false
  parallel stepsForParallel
} // stage

Answer 2

if you are using Jenkinsfile then, I achieved it via dynamically creating the stages, running them in parallel and also getting Jenkinsfile UI to show separate columns. This assumes parallel steps are independent of each other (otherwise don't use parallel) and you can nest them as deep as you want (depending upon the # of for loops you'll nest for creating stages).

Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section see here for more.

Answer 3

I use this to generate my stages which contain a Jenkins job in them. build_list is a list of Jenkins jobs that i want to trigger from my main Jenkins job, but have a stage for each job that is trigger.

build_list = ['job1', 'job2', 'job3']
        for(int i=0; i < build_list.size(); i++) {
          stage(build_list[i]){
               build job: build_list[i], propagate: false
          }
        }

Answer 4

Use the scripted syntax that allows more flexibility than the declarative syntax, even though the declarative is more documented and recommended.

For example stages can be created in a loop:

def tests = params.Tests.split(',')
for (int i = 0; i < tests.length; i++) {
    stage("Test ${tests[i]}") {
        sh '....'
    }
}

Answer 5

As JamesD suggested, you may create stages dynamically (but they will be sequential) like that:

def list
pipeline {
    agent none
    options {buildDiscarder(logRotator(daysToKeepStr: '7', numToKeepStr: '1'))}
    stages {
        stage('Create List') {
            agent {node 'nodename'}
            steps {
                script {
                    // you may create your list here, lets say reading from a file after checkout
                    list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
                }
            }
            post {
                cleanup {
                    cleanWs()
                }
            }
        }
        stage('Dynamic Stages') {
            agent {node 'nodename'}
            steps {
                script {
                    for(int i=0; i < list.size(); i++) {
                        stage(list[i]){
                            echo "Element: $i"
                        }
                    }
                }
            }
            post {
                cleanup {
                    cleanWs()
                }
            }
        }
    }
}

That will result in: dynamic-sequential-stages

Answer 6

@Jorge Machado: Because I cannot comment I had to post it as an answer. I've solved it recently. I hope it'll help you.

Declarative pipeline:

A simple static example:

stage('Dynamic') {
        steps {
            script {
                stage('NewOne') {

                        echo('new one echo')

                }
            }
        }
    }

Dynamic real-life example:

    // in a declarative pipeline
        stage('Trigger Building') {
              when {
                environment(name: 'DO_BUILD_PACKAGES', value: 'true')
              }
              steps {
                executeModuleScripts('build') // local method, see at the end of this script
              }
            }


    // at the end of the file or in a shared library
        void executeModuleScripts(String operation) {

          def allModules = ['module1', 'module2', 'module3', 'module4', 'module11']

          allModules.each { module ->  
          String action = "${operation}:${module}"  

          echo("---- ${action.toUpperCase()} ----")        
          String command = "npm run ${action} -ffffd"                   

            // here is the trick           
            script {
              stage(module) {
                bat(command)
              }
            }
          }

}