How to stop an unstoppable zombie job on Jenkins without restarting the server?

Question

Our Jenkins server has a job that has been running for three days, but is not doing anything. Clicking the little X in the corner does nothing, and the console output log do

Answer 1

Recently I came across a node/agent which had one executor occupied for days by a build "X" of a pipeline job, although that jobs page claimed build "X" did not exist anymore (discarded after 10 subsequent builds (!), as configured in the pipeline job). Verified that on disk: build "X" was really gone.

The solution: it was the agent/node which wrongly reported that the occupied executor was busy running build "X". Interrupting that executor's thread has immediately released it.

def executor = Jenkins.instance.getNode('NODENAME').computer.executors.find {
    it.isBusy() && it.name.contains('JOBNAME')
}

println executor?.name
if (executor?.isBusy()) executor.interrupt()

Other answers considered:

• The answer from @cheffe: did not work (see next point, and update below).
• The answers with Thread.getAllStackTraces(): no matching thread.
• The answer from @levente-holló and all answers with getBuildByNumber(): did not apply as the build wasn't really there anymore!
• The answer from @austinfromboston: that came close to my needs, but it would also have nuked any other builds running at the moment.

Update:
I experienced again a similar situation, where a Executor was occupied for days by a (still existing) finished pipeline build. This code snippet was the only working solution.

Answer 2

I had many zombi-jobs, so I used the following script:

for(int x = 1000; x < 1813; x = x + 1) {
    Jenkins .instance.getItemByFullName("JOBNAME/BRANCH")
    .getBuildByNumber(x)
    .finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"))
}

Answer 3

Go to "Manage Jenkins" > "Script Console" to run a script on your server to interrupt the hanging thread.

You can get all the live threads with Thread.getAllStackTraces() and interrupt the one that's hanging.

Thread.getAllStackTraces().keySet().each() {
  t -> if (t.getName()=="YOUR THREAD NAME" ) {   t.interrupt();  }
}

UPDATE:

The above solution using threads may not work on more recent Jenkins versions. To interrupt frozen pipelines refer to this solution (by alexandru-bantiuc) instead and run:

Jenkins.instance.getItemByFullName("JobName")
                .getBuildByNumber(JobNumber)
                .finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );

Answer 4

I use the Monitoring Plugin for this task. After the installation of the plugin

1. Go to Manage Jenkins > Monitoring of Hudson/Jenkins master
2. Expand the Details of Threads, the small blue link on the right side

3. Search for the Job Name that is hung

   The Thread's name will start like this

   Executor #2 for master : executing <your-job-name> #<build-number>

4. Click the red, round button on the very right in the table of the line your desired job has

Answer 5

I usually use jenkins-cli in such cases. You can download the jar from a page http://your-jenkins-host:PORT/cli . Then run

java -jar jenkins-cli.jar delete-builds name_of_job_to_delete hanging_job_number

Auxiliary info:

You may also pass a range of builds like 350:400. General help available by running

java -jar jenkins-cli.jar help

Context command help for delete-builds by

java -jar jenkins-cli.jar delete-builds

Answer 6

You can just copy the job and delete the old one. If it doesn't matter that you lost the old build logs.