Quartz Scheduler suddenly stop running and no exception error

Question

I have some quartz job which was running everyday at 7pm. Suddenly it failed to run. I check my server.log and there are no exception thrown. Anyone have any idea what could

Answer 1

I had similar but somewhat different problem. My scheduler is running fine in the development environment. In this scheduler I am doing the jobs like updating the transactions, etc.

When we move the build to production, the schedulers ran well and everything was fine until Saturday. On Saturday my scheduler was suddenly stopped. I didn't find any exception related to the scheduler in my app server (OC4J).

I am using quartz-1.5.2 version. I am unable to trace the actual root cause of the problem.

I am starting the scheduler on startup of the application server. If something goes wrong then it stops working. Then I am not having any chance to start them.

I think if I start the schedulers by calling the init servlet using some jsp request again makes the difference. It will be like seeing the profile (health of our schedulers and starting them again). If you have any better approach to start the scheduler then please suggest me.

Answer 2

Check if any Job is throwing an Exception. Put your Job exe code in a try catch block an trace any exception to troubleshoot the problem.

Answer 3

In my case I had an open connection to the database. When the I had no more connections available, my threads remained waiting forever. As I could not start any other jobs, nothing happened and everything stayed blocked. My advice is for you to check if you have any blocking resource that you might need to release.

Answer 4

I had a similar problem but the problem was, I had 10 threads quartz default number of threads in quartz properties and when I made thread dump* I found that I have 10 jobs in blocked stat, which means that I can't run any more threads.

A quick fix to this problem to increase the number of threads in the thread pool in the quartz properties.

The actual fix was reviewing my code to know why I had a 10 blocked threads.

*to do thread dump you can use kill -3 < java process number > which print the thread dump to your application standard output i.e if you running tomcat you find it in catalina.out log file

Answer 5

If you are using a database to store jobs, check the trigger_state of your trigger. Right now I'm seeing a similar problem (or at least it has similar symptoms).

A job that runs once a minute is leaving the trigger in "ACQUIRED" state and will never run again. Like you I'm seeing nothing the the log.

I'm also seeing a different cause of the same problem. Again, the job just stops running, but the trigger is not in the "ACQUIRED" state. So far I don't know the cause.

What I know so far is that the scheduler thread is waiting for a free worker thread. It looks like all of the worker threads are waiting for a semaphore in order to update their schedule. I haven't been able to get a thread dump yet to verify what the worker threads are waiting on.

I'm running Quartz 1.6.1 RC1. See this bug report: http://jira.opensymphony.com/browse/QUARTZ-668

I think that's what I'm seeing.

Answer 6

I had a similar problem and unfortunately, none of the above answers helped me to figure out my problem. I am using quartz-2.3.2 version. First of all, i agree with some others, wherein most of the cases the scheduler not fired up, is caused due to threads race conditions which blocked and try to acquire the flag from the thread entered the critical zone and it does not release it. This simply smells a kind of bad code but anyway, i don't want to say again the same thing like the others did.I want to come up with my solution to give you the way on which i fixed the problem.

Assuming you use a Cron scheduler like the following way i will provide a detailed example

this is the SimpleJob class

import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.JobKey;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Date;

public class SimpleJob implements Job {

    private static Logger _log = LoggerFactory.getLogger(SimpleJob.class);

    public SimpleJob() {
    }

    public void execute(JobExecutionContext context)
            throws JobExecutionException {

        // This job simply prints out its job name and the
        // date and time that it is running
        JobKey jobKey = context.getJobDetail().getKey();
        _log.info("SimpleJob says: " + jobKey + " executing at " + new Date());
    }

}

This is your main class

import org.quartz.CronTrigger;
import org.quartz.JobDetail;
import org.quartz.Scheduler;
import org.quartz.impl.StdSchedulerFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import static org.quartz.CronScheduleBuilder.cronSchedule;
import static org.quartz.JobBuilder.newJob;
import static org.quartz.TriggerBuilder.newTrigger;

public class CronTriggerExample {

    public void run() throws Exception {
        Logger log = LoggerFactory.getLogger(CronTriggerExample.class);

        log.info("------- Initializing -------------------");

        // First we must get a reference to a scheduler
        JobDetail job = newJob(SimpleJob.class)
                .withIdentity("job1", "group1").build();

        CronTrigger trigger = newTrigger()
            .withIdentity("trigger1", "group1")
            .withSchedule(cronSchedule("10 10/5 * ? * *").withMisfireHandlingInstructionFireAndProceed())
            .build();
        Scheduler scheduler = new StdSchedulerFactory().getScheduler();
        scheduler.start();
        scheduler.scheduleJob(job, trigger);
    }

    public static void main(String[] args) throws Exception {

        CronTriggerExample example = new CronTriggerExample();
        example.run();
    }
}

The below cron expression means At second :10, every 5 minutes starting at minute :10, of every hour.

cronSchedule("10 10/5 * ? * *")

If you notice the most interesting part here is the

withMisfireHandlingInstructionFireAndProceed()

This is the key to solve your problem if your trigger is felt in a misfire situation, it instructs your scheduler to be fired immediately. An alternative occasion is to use

withMisfireHandlingInstructionDoNothing()

where upon a mis-fire situation the cronTrigger will be next fired at the started time which the scheduler is set on. For example in our case in At second :10, every 5 minutes starting at minute :10, of every hour.