What's the best way to count results in GQL?

Question

I figure one way to do a count is like this:

foo = db.GqlQuery("SELECT * FROM bar WHERE baz = 'baz')
my_count = foo.count()

What I don't like is my count will be limited to 1000 max and my query will probably be slow. Anyone out there with a workaround? I have one in mind, but it doesn't feel clean. If only GQL had a real COUNT Function...

Answer 1

+1 to Jehiah's response.

Official and blessed method on getting object counters on GAE is to build sharded counter. Despite heavily sounding name, this is pretty straightforward.

Answer 2

You have to flip your thinking when working with a scalable datastore like GAE to do your calculations up front. In this case that means you need to keep counters for each baz and increment them whenever you add a new bar, instead of counting at the time of display.

class CategoryCounter(db.Model):
    category = db.StringProperty()
    count = db.IntegerProperty(default=0)

then when creating a Bar object, increment the counter

def createNewBar(category_name):
  bar = Bar(...,baz=category_name)

  counter = CategoryCounter.filter('category =',category_name).get()
  if not counter:
    counter = CategoryCounter(category=category_name)
  else:
    counter.count += 1
  bar.put()
  counter.put()

db.run_in_transaction(createNewBar,'asdf')

now you have an easy way to get the count for any specific category

CategoryCounter.filter('category =',category_name).get().count

Answer 3

Count functions in all databases are slow (eg, O(n)) - the GAE datastore just makes that more obvious. As Jehiah suggests, you need to store the computed count in an entity and refer to that if you want scalability.

This isn't unique to App Engine - other databases just hide it better, up until the point where you're trying to count tens of thousands of records with each request, and your page render time starts to increase exponentially...

Answer 4

According to the GqlQuery.count() documentation, you can set the limit to be some number greater than 1000:

from models import Troll
troll_count = Troll.all(keys_only=True).count(limit=31337)

Sharded counters are the right way to keep track of numbers like this, as folks have said, but if you figure this out late in the game (like me) then you'll need to initialize the counters from an actual count of objects. But this is a great way to burn through your free quota of Datastore Small Operations (50,000 I think). Every time you run the code, it will use up as many ops as there are model objects.

Answer 5

I haven't tried it, and this is an utter resource hog, but perhaps iterating with .fetch() and specifying the offset would work?

LIMIT=1000
def count(query):
   result = offset = 0
   gql_query = db.GqlQuery(query)
   while True:
     count = gql_query.fetch(LIMIT, offset)
     if count < LIMIT:
       return result
     result += count
     offset += LIMIT

Answer 6

orip's solution works with a little tweaking:

LIMIT=1000
def count(query):
    result = offset = 0
    gql_query = db.GqlQuery(query)
    while True:
        count = len(gql_query.fetch(LIMIT, offset))
        result += count
        offset += LIMIT
        if count < LIMIT:
            return result

Answer 7

We now have Datastore Statistics that can be used to query entity counts and other data. These values do not always reflect the most recent changes as they are updated once every 24-48 hours. Check out the documentation (see link below) for more details:

Datastore Statistics

Answer 8

As pointed out by @Dimu, the stats computed by Google on a periodic basis are a decent go-to resource when precise counts are not needed and the % of records are NOT changing drastically during any given day.

To query the statistics for a given Kind, you can use the following GQL structure:

select * from __Stat_Kind__ where kind_name = 'Person'

There are a number of properties returned by this which are helpful:

• count -- the number of Entities of this Kind
• bytes -- total size of all Entities stored of this Kind
• timestamp -- an as of date/time for when the stats were last computed

Example Code

To answer a follow-up question posted as a comment to my answer, I am now providing some sample C# code that I am using, which admittedly may not be as robust as it should be, but seems to work OK for me:

/// <summary>Returns an *estimated* number of entities of a given kind</summary>
public static long GetEstimatedEntityCount(this DatastoreDb database, string kind)
{
    var query = new GqlQuery
    {
        QueryString = $"select * from __Stat_Kind__ where kind_name = '{kind}'",
        AllowLiterals = true
    };
    var result = database.RunQuery(query);
    return (long) (result?.Entities?[0]?["count"] ?? 0L);
}

Answer 9

The best workaround might seem a little counter-intuitive, but it works great in all my appengine apps. Rather than relying on the integer KEY and count() methods, you add an integer field of your own to the datatype. It might seem wasteful until you actually have more than 1000 records, and you suddenly discover that fetch() and limit() DO NOT WORK PAST THE 1000 RECORD BOUNDARY.

def MyObj(db.Model):
  num = db.IntegerProperty()

When you create a new object, you must manually retrieve the highest key:

max = MyObj.all().order('-num').get()
if max : max = max.num+1
else : max = 0
newObj = MyObj(num = max)
newObj.put()

This may seem like a waste of a query, but get() returns a single record off the top of the index. It is very fast.

Then, when you want to fetch past the 1000th object limit, you simply do:

MyObj.all().filter('num > ' , 2345).fetch(67)

I had already done this when I read Aral Balkan's scathing review: http://aralbalkan.com/1504 . It's frustrating, but when you get used to it and you realize how much faster this is than count() on a relational db, you won't mind...