Django REST Framework: slow browsable UI because of large related table

I have a model in my API that has a foreign key to a table with tens of thousands of records. When I browse to that model's detail page in the browsable UI, the page load takes forever because it is trying to populate the foreign key dropdown with tens of thousands of entries for the HTML form for the PUT command.

Is there anyway to work around this? I think my best solution would be to have the browsable UI not show this field and thus prevent the slow load. People can still update the field by an actual PUT api request directly.

Thanks.

Tom Christie

Take a look at using an autocomplete widget, or drop down to using a dumb textfield widget.

Autocompletion docs here: http://www.django-rest-framework.org/topics/browsable-api/#autocomplete

You can force using TextInput with simple:

from django.forms import widgets
...
class YourSerializer(serializers.ModelSerializer):
    param = serializers.PrimaryKeyRelatedField(
        widget=widgets.TextInput
    )

Or after proper autocomplete_light configuration:

import autocomplete_light
...
class YourSerializer(serializers.ModelSerializer):
    paramOne = serializers.PrimaryKeyRelatedField(
        widget=autocomplete_light.ChoiceWidget('RelatedModelAutocomplete')
    )
    paramMany = serializers.PrimaryKeyRelatedField(
        widget=autocomplete_light.MultipleChoiceWidget('RelatedModelAutocomplete')
    )

To filter out results which are returned by autocomplete_light this part of documentation.

Note that you can disable the HTML form and keep the raw data json entry with:

class BrowsableAPIRendererWithoutForms(BrowsableAPIRenderer):
    """Renders the browsable api, but excludes the forms."""
    def get_rendered_html_form(self, data, view, method, request):
        return None

and in settings.py:

REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
        'application.api.renderers.BrowsableAPIRendererWithoutForms',
    ),
}

this will speed things up and you can still post from from the browsable ui.

It is a very good question for none obvious problem. The fault assumptions that you are suck into while learning Django, and related to it plugins DRF while reading official documentation, will create a conceptual model that is simply not true. I'm talking here about, Django being explicitly design for relational databases, does not make it fast out of the box!

Problem

The reason for Django/DRF being slow while querying a model that contain relationships (e.g. one-to-many) in ORM world is known as N+1 problem (N+1, N+1) and is particularity noticeable when ORM uses lazy loading - Django uses lazy loading !!!

Example

Let's assume that you have a model that looks like this: a Reader has many Books. Now you would like to fetch all books 'title' read by 'hardcore' reader. In Django you would execute this by interacting with ORM in this way.

# First Query: Assume this one query returns 100 readers.
> readers = Reader.objects.filter(type='hardcore')

# Constitutive Queries
> titles = [reader.book.title for reader in readers]

Under the hood. The first statement Reader.objects.filter(type='hardcore') would create one SQL query that looks similar to that. We assume that it will return 100 records.

SELECT * FROM "reader" WHERE "reader"."type" = "hardcore";

Next, for each reader [reader.book.title for reader in readers] you would fetch related books. This in SQL would look similar to that.

SELECT * FROM "book" WHERE "book"."id" = 1;
SELECT * FROM "book" WHERE "book"."id" = 2;
...
SELECT * FROM "book" WHERE "book"."id" = N;

What you left with is, 1 select to fetch 100 readers, and N selects to get books -where N is number of books. So in total you have N+1 queries against database.

Consequence of this behavior is 101 queries against database, that at the end results with extremely long loading time for small amount of data and making Django slow!

Solution

Solution is easy but not obvious. Following official documentation for Django or DRF does not highlight the problem. At the end you follow the best practices and you end up with slow application.

To fix slow loading problem you will have to eager load your data in Django. Usually, this mean using appropriate prefetch_related() or select_related() method to construct SQL INNER JOIN on models/tables, and fetch all your data just in 2 queries instead of 101.

Django REST Framework: slow browsable UI because of large related table

Problem

Example

Solution

Related Reads