Django Celery Database for Models on Producer and Worker

Question

I want to develop an application which uses Django as Fronted and Celery to do background stuff. Now, sometimes Celery workers on different machines need database access to my

Answer 1

They will need access to the database. That access will be through a database backend, which can be one that ships with Django or one from a third party.

One thing I've done in my Django site's settings.py is load database access info from a file in /etc. This way the access setup (database host, port, username, password) can be different for each machine, and sensitive info like the password isn't in my project's repository. You might want to restrict access to the workers in a similar manner, by making them connect with a different username.

You could also pass in the database connection information, or even just a key or path to a configuration file, via environment variables, and handle it in settings.py.

For example, here's how I pull in my database configuration file:

g = {}
dbSetup = {}
execfile(os.environ['DB_CONFIG'], g, dbSetup)
if 'databases' in dbSetup:
    DATABASES = dbSetup['databases']
else:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            # ...
        }
    }

Needless to say, you need to make sure that the file in DB_CONFIG is not accessible to any user besides the db admins and Django itself. The default case should refer Django to a developer's own test database. There may also be a better solution using the ast module instead of execfile, but I haven't researched it yet.

Another thing I do is use separate users for DB admin tasks vs. everything else. In my manage.py, I added the following preamble:

# Find a database configuration, if there is one, and set it in the environment.
adminDBConfFile = '/etc/django/db_admin.py'
dbConfFile = '/etc/django/db_regular.py'
import sys
import os
def goodFile(path):
    return os.path.isfile(path) and os.access(path, os.R_OK)
if len(sys.argv) >= 2 and sys.argv[1] in ["syncdb", "dbshell", "migrate"] \
    and goodFile(adminDBConfFile):
    os.environ['DB_CONFIG'] = adminDBConfFile
elif goodFile(dbConfFile):
    os.environ['DB_CONFIG'] = dbConfFile

Where the config in /etc/django/db_regular.py is for a user with access to only the Django database with SELECT, INSERT, UPDATE, and DELETE, and /etc/django/db_admin.py is for a user with these permissions plus CREATE, DROP, INDEX, ALTER, and LOCK TABLES. (The migrate command is from South.) This gives me some protection from Django code messing with my schema at runtime, and it limits the damage an SQL injection attack can cause (though you should still check and filter all user input).

This isn't a solution to your exact problem, but it might give you some ideas for ways to smarten up Django's database access setup for your purposes.