Annotate with value of latest related in Django 1.8 using conditional annotation

Question

I have the following models:

class City(models.Model):
    ...

class Census(models.Model):
    city = models.ForeignKey(City)
    date = models.DateTimeField()
    value = models.BigIntegerField()

Now I'd like to annotate a City-queryset with the value of the latest Census. How do I achieve that?

I have tried:

City.objects.annotate(population=Max('census__date'))
# --> annotates date and not value

City.objects.annotate(population=Max('census__value'))
# --> annotates highest value, not latest

City.objects.annotate(population=
    Case(
        When(
            census__date=Max('census__date'),
            then='census__value')
        )
    )

# --> annotates 'None'

City.objects.annotate(population=
    Case(
        When(
            census__date=Max('census__date'),
            then='census__value')
        ), output_field=BigIntegerField()
    )

# --> takes forever (not sure what happens at the end, after some minutes I stopped waiting)

Any help greatly appreciated!

Answer 1

I've also been having an issue where I need a max value object of a related set, but I require the entire object. I have not been able to figure out a solution using annotation and Case. Until I do, I use this prefetching solution. If each city does not have a large amount of census objects, or if your application is not performance bound, this may work for you.

inner_qs = Census.objects.order_by('-date')
cities = City.objects.prefetch_related(Prefetch("census_set", queryset=inner_qs, to_attr="census_list"))

class City(models.Model):
    @property
    def latest_census(self):
        if hasattr(self, 'census_list') and len(self.census_list) > 0:
            return self.census_list[0]
        return None

If this does not work for you, consider some of the suggestions found here: http://blog.roseman.org.uk/2010/08/14/getting-related-item-aggregate/

Answer 2

At this moment, they are not a django query expression to annotate a not aggregated field from a related 1:N model based on a sql having expression.

You can accomplish it with several workarounds like split query and work with data in memory ( itertools groupby f.e. ) or through raw queries. But this will not match your requirements performance and database agnostic.

I explain here what I will do if this was my app. For developers is hard to have redundancy in database. In your scenario, last census by city is a calculated field ... but, in this case, take in consideration to materialize last_census:

The dirty work ...

class City(models.Model):
    last_census = models.ForeignKey('Census', null=True, 
                                     blank=True, editable=False)
    ...

For easy maintenance, you can overrite save and delete methods on Census to keep last_census up to date.

class Census(models.Model):
    ...

    #overriding save
    def save(self, *args, **kwargs):
        super(Census, self).save(*args, **kwargs)
        #updating last_census on referenced city
        max_census = Census.objects.filter( city = self.city ).latest('date')
        self.city.last_census = max_census.city if max_census else None
        self.city.save()

    #overriding delete
    def delete(self, *args, **kwargs):
        #updating last_census on referenced city
        max_census = ( Census.objects
                      .filter( city = self.city )
                      .exclude( id = self.id )
                      .latest('date') )
        self.city.last_census = max_census.city if max_census else None
        self.city.save()
        super(Census, self).delete(*args, **kwargs)

Notice: if you are more comfortable, you can code it with signals ( pre_delete, post_save, ... ) instead overriding methods.

The best ...

Your query now:

City.objects.select_related('last_census__value').all()

Answer 3

Something like this may work for you:

I have this to show last reservation for restaurants

Reservation.objects.filter(
        restaurant__city_id__gt=0
        ).distinct().annotate(city_count=Count(
        'restaurant_id')
        ).order_by('-reservationdate').filter(city_count__gte=1)[:20]

in your case it may be something like:

city = Census.objects.filter(
        city_id__gt=0
        ).distinct().annotate(city_count=Count(
        'city_id')
        ).order_by('-date').filter(city_count__gte=1)[:20]

and your html

{% for city in city %}
{{ city.city.name }} {{ city.date }}<br>
{% endfor %}

Answer 4

It's late and I'm getting hungry so I see won't this all the way through*, but the following code will return the latest census value in a valueset for a given city. A queryset might be possible, but again I am hungry!

*I wasn't sure if you needed to retrieve all of the latest values for all cities or a specific city. The latter is pretty easy, the former is a bit harder. You could easily put this as a method on your model and call it on every city in a loop inside of a template/view.

Hope this helps!

from app.models import *
from django.db.models import F

    City.objects.annotate(
        values=F("census__value"),
        date=F("census__date"))\
        .values('values', 'name').filter(name="Atlanta")\
        .latest('date')