Celery logger configuration

Question

I\'m using Django 1.10, python 3.5 and celery 4.1.0 I\'m trying to log celery tasks info into a file. So I tried as suggested in celery documentation -

from cele         


        

Answer 1

For what it's worth, this is how I configured celery to use my Django logging settings:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.signals import setup_logging

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')

app = Celery('app')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

@setup_logging.connect
def config_loggers(*args, **kwags):
    from logging.config import dictConfig
    from django.conf import settings
    dictConfig(settings.LOGGING)

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

That was about it - after that change my django logging settings worked for celery, including the logging I had setup to send log messages over to slack.

Answer 2

By default, celery will reset handers on celery.task logger, you could disable this behavior with worker_hijack_root_logger option. Or, you could reconfigure this logger in after_setup_task_logger signal, even dont let celery config the loggers with setup_logging signal:

from celery.signals import setup_logging

@setup_logging.connect()
def config_loggers(*args, **kwargs):
    from logging.config import dictConfig
    dictConfig(app.config['LOGGING_CONFIG'])

Answer 3

I was fiddling my logging this morning trying to figure out why my log was not progagated to the root logger, turned out as what @georgexsh said, celey is hijacking the root logger, here is my working logger config:

'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'cal': {
            'handlers': ['general'],
            'level': 'INFO',
            'propagate': True,
        },
        '': {
            'handlers': ['gelf'],
            'level': 'INFO',
        }
    }

And with CELERYD_HIJACK_ROOT_LOGGER = False in the settings.

All the celery logs are now going to graylog(defined in root) instead of the usual place.