Quartz.NET scheduler doesn't fire jobs/triggers once deployed

Question

INTRODUCTION

I'm using Quartz.Net on an ASP.Net framework 4, webforms web site. Basically, the user should have the hability to fire manually a batch script that asynchronously processes thousands of records stored on a database. The user can stop or pause at any time, adjust some variables, and continue if needed with the process (remaining records).
The code is done and working locally (developer machine, win7, vs2010, sql server express 2008 R2).
It was also tested on a local server (win server 2008 R2, sql server express 2008 R2). It works fine on both enviroments, tested with all the code precompiled. The problem is that, once deployed on a remote server (win server 2008 R2), where it actually should be running on (hosting enviroment, not shared, not clustered), it doesnt completely works (se details below). Scheduler gets created, but the trigger, hence the job, doesn't fire.

(Note: I know some of you would suggest to use Quartz as windows service, but despite the benefits of doing so, I really would like to find out why it doesn't work as an embedded solution, since it should be working just fine like does locally)

DETAILS

Quartz 2.1.2  
Common.Logging 2.1.2  
Common.Logging.NLog 2.0.0  
NLog 2.0.1.2

global.asax

public static ISchedulerFactory SchedulerFactory;
public static IScheduler Scheduler;

void Application_Start(object sender, EventArgs e)
{
    SchedulerFactory = new StdSchedulerFactory();
    Scheduler = SchedulerFactory.GetScheduler();

    // Define a durable job instance (durable jobs can exist without triggers)
    IJobDetail job = JobBuilder.Create<MyJobClass>()
                                .WithIdentity("MyJob", "MyGroup")
                                .StoreDurably()
                                .Build();

    Scheduler.AddJob(job, false);
    Scheduler.Start();
}
void Application_End(object sender, EventArgs e)
{
    Scheduler.Shutdown(true);
}

process.aspx.cs (start button click)

// get records from DB, iterate, process, etc
...

IJobDetail job = ASP.global_asax.Scheduler.GetJobDetail(new JobKey("MyJob", "MyGroup"));
job.JobDataMap.Put("something1", 1);
job.JobDataMap.Put("something2", somevar);

ITrigger trigger = TriggerBuilder.Create()
                    .WithIdentity("MyTrigger", "MyGroup")
                    .StartNow()
                    .WithSimpleSchedule(x => x.WithIntervalInSeconds(5).RepeatForever())
                    .Build();

var triggersSet = new Quartz.Collection.HashSet<ITrigger> { trigger };

ASP.global_asax.Scheduler.ScheduleJob(job, triggersSet, true);

LOG OUTPUT

local log

Default Quartz.NET properties loaded from embedded resource file  
Using default implementation for object serializer  
Using default implementation for ThreadExecutor  
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl  
Quartz Scheduler v.2.1.2.400 created.  
RAMJobStore initialized.  
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'   Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally.   NOT STARTED.   Currently in standby mode.   Number of jobs executed: 0   Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads.   Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.  
Quartz scheduler 'DefaultQuartzScheduler' initialized  
Quartz scheduler version: 2.1.2.400  
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.  
Batch acquisition of 0 triggers  
Batch acquisition of 0 triggers

It continues logging Batch acquisition of 0 triggers until button click occurs:

Default Quartz.NET properties loaded from embedded resource file  
Batch acquisition of 1 triggers  
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass  
Batch acquisition of 0 triggers  
Calling Execute on job MyGroup.MyJob  
Trigger instruction : NoInstruction  
Batch acquisition of 1 triggers  
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass  
Batch acquisition of 0 triggers  
Calling Execute on job MyGroup.MyJob  
Trigger instruction : NoInstruction  
Batch acquisition of 1 triggers

deployed log

Default Quartz.NET properties loaded from embedded resource file  
Using default implementation for object serializer  
Using default implementation for ThreadExecutor  
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl  
Quartz Scheduler v.2.1.2.400 created.  
RAMJobStore initialized.  
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads. Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.   
Quartz scheduler 'DefaultQuartzScheduler' initialized  
Quartz scheduler version: 2.1.2.400  
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.

Here stays like this. As you see, compared to the other log, it's not trying to acquire triggers (line Batch acquisition of 0 triggers does not appear at all). If you click the process button anyway, the log adds one line:

Default Quartz.NET properties loaded from embedded resource file

But nothing else happens. The records are not processed (I know since every time a record is proccessed, is marked in the database). No errors occur, but the trigger is not fired, and the job is not executed. Also, the CPU usage run up to 50% or more on button click, and doesnt gets down unless you go to IIS, stop and restart the application pool. This cpu consumption doesn't happen locally.

---

update 1

Changed use of scheduler for a singleton, as suggested by LeftyX, but still get same behavior on remote server.

update 2

I also tried to use ADOJobStore (instead of RAMJobStore which I was using). Now it still works perfectly locally; but still doesn't execute the trigger (hence job) online. The only difference is that online the CPU usage doesn't run up to 50%. And now I can see that the job and trigger are created (I query the tables and see that those records exists), but never gets executed.

Answer 1

One thing that I have noticed is the use of the Scheduler in your asp.net application.
You should use singleton objects.

in your process.aspx.cs this line

IScheduler scheduler = new StdSchedulerFactory().GetScheduler();

creates a new scheduler but you should use the one you've created as static in Application_Start.

If you want to get access to the singleton instance use a public memeber in your Global.asax.cs:

 public static ISchedulerFactory SchedulerFactory;
 public static IScheduler Scheduler;

and you can reference it in your process.aspx.cs:

MvcApplication.Scheduler.ScheduleJob(job, triggersSet, true);

Another solution is to use dependency injection. You can find some info here using StructureMap and here for Unity.

UPDATE:

You can download a sample application (asp.net 4.0) called AspNet_Quartz here and see how it works here.

Answer 2

The problem is related to IIS rather than the schedulers Quartz.NET, Hangfire, etc. On the other hand, there are lots of solution methods posted on the web, but only some of them is working. In my opinion, there is no need to apply lots of configuration settings. Just install Keep Alive Service For IIS 6.0/7.5 on the server to which you publish your application and enjoy. After that, your published application will be alive after application pool recycling, IIS/Application restarting, etc. Hope this helps...

Answer 3

There are nothing wrong with Quartz, all because of IIS app pool recycling. I fixed the bug by stopping the pool that is used for Quartz from recycling:
1. Go to IIS manager -> Application Pools -> Create a new pool, I named it Scheduler (u can named anything)
2. Select Scheduler pool -> advanced Settings
+ In General section, at Start Mode, Select AlwaysRunning (IIS 8.5) or true for (IIS 7.5, 8)
+ In Process Model Section-> Idle Timeout(minutes) set to 0 (meaning: No Idel timeout)
+ In Recycling section -> Regular time Interval set to 0 (meaning: no recycling)
3. Deploy your Quartz site into that application pool. And send one request to the pool to "wake your app up" and it will run until u stop it.

That's it.
Updated: Another solution to keep your app pool always alive is using Auto-Start ASP.NET Applications

Answer 4

I just ran into a similar issue that might bite someone else - it brought me to this SO question after turning on debug and getting that 'batch acquisition of 0 triggers' message.
I had a job that was every 2 hours like this:

"0 0 0/2 * * ?"

And I wanted to make it more often, so every 2 minutes like this:

"0 0/2 0 * ?"

I even tried https://cronexpressiondescriptor.azurewebsites.net/ which gave me a big clue I should have read more carefully: Every 2 hours, on day 0 of the month which eventually made me realize what I really meant was:

"0 0/2 * * ?"

So the lesson was, when 'shifting left' your cron, back-fill with *, not 0.

Hope that helps someone else.