Django Unit Testing taking a very long time to create test database

Question

For some time now, my unit testing has been taking a longer than expected time. I have tried to debug it a couple of times without much success, as the delays are before my

Answer 1

The final solution that fixes my problem is to force Django to disable migration during testing, which can be done from the settings like this

TESTING = 'test' in sys.argv[1:]
if TESTING:
    print('=========================')
    print('In TEST Mode - Disableling Migrations')
    print('=========================')

    class DisableMigrations(object):

        def __contains__(self, item):
            return True

        def __getitem__(self, item):
            return "notmigrations"

    MIGRATION_MODULES = DisableMigrations()

or use https://pypi.python.org/pypi/django-test-without-migrations

My whole test now takes about 1 minute and a small app takes 5 seconds.

In my case, migrations are not needed for testing as I update tests as I migrate, and don't use migrations to add data. This won't work for everybody

Answer 2

Database initialization indeed takes too long...

I have a project with about the same number of models/tables (about 77), and approximately 350 tests and takes 1 minute total to run everything. Deving in a vagrant machine with 2 cpus allocated and 2GB of ram. Also I use py.test with pytest-xdist plugin for running multiple tests in parallel.

Another thing you can do is tell django reuse the test database and only re-create it when you have schema changes. Also you can use SQLite so that the tests will use an in-memory database. Both approaches explained here: https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database

EDIT: In case none of the options above work, one more option is to have your unit tests inherit from django SimpleTestCase or use a custom test runner that doesn't create a database as explained in this answer here: django unit tests without a db.

Then you can just mock django calls to the database using a library like this one (which admittingly I wrote): https://github.com/stphivos/django-mock-queries

This way you can run your unit tests locally fast and let your CI server worry about running integration tests that require a database, before merging your code to some stable dev/master branch that isn't the production one.

Answer 3

Summary

Use pytest !

Operations

1. pip install pytest-django
2. pytest --nomigrations instead of ./manage.py test

Result

• ./manage.py test costs 2 min 11.86 sec
• pytest --nomigrations costs 2.18 sec

Hints

• You can create a file called pytest.ini in your project root directory, and specify default command line options and/or Django settings there.

  # content of pytest.ini
  [pytest]
  addopts = --nomigrations
  DJANGO_SETTINGS_MODULE = yourproject.settings
  

  Now you can simply run tests with pytest and save you a bit of typing.

• You can speed up the subsequent tests even further by adding --reuse-db to the default command line options.

  [pytest]
  addopts = --nomigrations --reuse-db
  

  However, as soon as your database model is changed, you must run pytest --create-db once to force re-creation of the test database.

• If you need to enable gevent monkey patching during testing, you can create a file called pytest in your project root directory with the following content, cast the execution bit to it (chmod +x pytest) and run ./pytest for testing instead of pytest:

  #!/usr/bin/env python
  # -*- coding: utf-8 -*-
  # content of pytest
  from gevent import monkey
  
  monkey.patch_all()
  
  import os
  
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourproject.settings")
  
  from django.db import connection
  
  connection.allow_thread_sharing = True
  
  import re
  import sys
  
  from pytest import main
  
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(main())
  

  You can create a test_gevent.py file for testing whether gevent monkey patching is successful:

  # -*- coding: utf-8 -*-
  # content of test_gevent.py
  import time
  from django.test import TestCase
  from django.db import connection
  import gevent
  
  
  def f(n):
      cur = connection.cursor()
      cur.execute("SELECT SLEEP(%s)", (n,))
      cur.execute("SELECT %s", (n,))
      cur.fetchall()
      connection.close()
  
  
  class GeventTestCase(TestCase):
      longMessage = True
  
      def test_gevent_spawn(self):
          timer = time.time()
          d1, d2, d3 = 1, 2, 3
          t1 = gevent.spawn(f, d1)
          t2 = gevent.spawn(f, d2)
          t3 = gevent.spawn(f, d3)
          gevent.joinall([t1, t2, t3])
          cost = time.time() - timer
          self.assertAlmostEqual(cost, max(d1, d2, d3), delta=1.0,
                                 msg='gevent spawn not working as expected')
  

References

• pytest-django documentation
• pytest documentation

Answer 4

use ./manage.py test --keepdb when there are no changes in the migration files