Jenkins/Hudson - How to run multiple jobs in parallel more than 1 level deep?

Question

I'm trying to get the following workflow running in Jenkins, in parallel. So for example, both A and B running at the same time. As soon as A finishes run A2 and A3 at the same time, etc...

It seems like a pretty common use case but most plugins I tried do no support more than 1 level deep, which is the case with the A branch below. Join plug-in doesn't seem helpful here.

I read about using the Promotion plugin but I'm a little mystified on what to fingerprint/artifacts to archive to make this work.

Any clue on how to make this simple build pipeline work?

Answer 1

As jgritty pointed out you could use the Build flow plugin. In order to get the type of parallel execution you want you could run something equivalent to the following build flow script:

buildTrigger = build("Trigger")

parallel(
    {
        buildA = build("A")
        buildA1 = build("A1")
        parallel(
            {
                buildA2 = build("A2")
            },
            {
                buildA3 = build("A3")
            },
        )
    },
    {
        buildB = build("B")
        buildB1 = build("B1")
    },
)

buildResults = build("GatherResult")

In this script the first parallel block takes care of the A and B branches. Once in a branch each build is sequential until you add more parallel blocks.

Answer 2

There is a solution which works great: the Multi-Job plugin. It does exactly what you want. With The MultiJob Plugin you can split your job into phases which run serially. Within each phase, the jobs run in parallel.

For simplicity I'm going to assume that A and B are "compilation jobs" you wish to run in parallel. Further assume there may be an A-Test (and its children) and B-Test which are also separate jobs.

You create the multijob as follows: New Item -> Multijob Project

In the project you create two phases (Add build step "MultiJob Phase"). The first will be COMPILE and you would add Phase Job's "A" and "B".

You can change the options for the COMPILE phase jobs so that if either fails, the whole phase aborts (the default), or allow the job to continue.

Next you add another MultiJob Phase Build step for TEST and add "A-Test" and "B-Test" to that. Remember that A, B, A-Test, and B-Test are all separate jobs that can also be run individually if desired.

That's it. When the job runs it will contain links to the child jobs so you can see what happened with the sub-jobs.

Answer 3

I believe this may be the plugin you need.

Answer 4

I had a similar problem, and the Promoted Builds Plugin was a better solution for me than the Build Flow Plugin that others have advised.

A build promotion job can be added to an existing job by ticking the check-box "Promote builds when...". You can configure the promotion to occur when named downstream projects are complete by checking "When the following downstream projects build successfully" under Criteria (assuming that all downstream projects are related by fingerprints).

At the promotion step, you can trigger emails of summary results or trigger a further downstream job.

In this case, you might add the promotion job to the Trigger projects, and depend on projects A2 A3 and B1 completing successfully.

Answer 5

Use Build Other projects in Post-Build Section. Say start A2,A3 on completion of A1. Increase number of executors in Manage Hudson --> Configure system to appropriate number.

Answer 6

We have a set up similar to what 'Steven the Easily Amused' suggested. Except I set up the "A" and "B" paths as their own Multi-Job stack and another Multi-Job set up as the "Trigger". Where both the "A" and "B" stack run in parallel. This way they don't directly have a tight dependency on each other. You can also aggregate your results depending on if one/both stacks fail. You can also set up the "Trigger" to collect parameters and pass then to both stacks.