when using laravel queue job processing, getting stuck

Question

i am on 5.3.31

so it cannot be related to

https://github.com/laravel/framework/issues/15179

after 300ish jobs I get:

[2017-04-11 13:51:53] local.ERROR: Symfony\Component\Debug\Exception\FatalThrowableError: Call to a member function beginTransaction() on null in /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Database/Connection.php:612
Stack trace:
#0 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(175): Illuminate\Database\Connection->beginTransaction()
#1 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(175): Illuminate\Queue\DatabaseQueue->pop('default')
#2 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), 'default')
#3 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('database', 'default', Object(Illuminate\Queue\WorkerOptions))
#4 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions))
#5 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default')
#6 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#7 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Container/Container.php(508): call_user_func_array(Array, Array)
#8 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array)
#9 /var/www/html/www.myapp.com/vendor/symfony/console/Command/Command.php(256): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(820): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(187): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(118): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/www.myapp.com/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 {main}  

this error keeps repeating

the jobs are stuck, nothing advances, nothing gets moved to failed-queue neither

the only solution is that I have found is stopping the supervisor workers via cronjob every couple minutes, and starting them again. Which is far from optimal

seems as well as if the whole server disappears once in a while quickly. Maybe a memory problem? i have 1 gb of memory for my live server.

Answer 1

It is probably one of two things:

1. Memory Leak

If you are running your queues as a daemon process, then it is possible that over time they will run out of memory. If this occurs, you have three choices.

• Find a stop the memory leak
• Switch to using queue:listen to ensure PHP restarts each time
• Run a scheduler that includes $schedule->command('queue:restart')->hourly(). This gives you the best of both worlds, because you get the benefit of a daemon process, and just restart it once an hour to allow PHP to reset.

2. Failed database connection

There is currently a known issue that if a DB connection goes away during a daemon process, it may not reconnect. A PR is in process here - so check if this gets accepted and the bug should be fixed: https://github.com/laravel/framework/pull/19080

Monitoring

If you want to know your queues are always working, you can use a service like Eyewitness.io. This is a dedicated Laravel application monitoring, that includes queue monitoring. If your queues stop working for any reason - you'll get notified. (Disclaimer: it is my site)