SQLite: COUNT slow on big tables

Question

I\'m having a performance problem in SQLite with a SELECT COUNT(*) on a large tables.

As I didn\'t yet receive a usable answer and I did some further testing, I edited m

Answer 1

Here's a potential workaround to improve the query performance. From the context, it sounds like your query takes about a minute and a half to run.

Assuming you have a date_created column (or can add one), run a query in the background each day at midnight (say at 00:05am) and persist the value somewhere along with the last_updated date it was calculated (I'll come back to that in a bit).

Then, running against your date_created column (with an index), you can avoid a full table scan by doing a query like SELECT COUNT(*) FROM TABLE WHERE date_updated > "[TODAY] 00:00:05".

Add the count value from that query to your persisted value, and you have a reasonably fast count that's generally accurate.

The only catch is that from 12:05am to 12:07am (the duration during which your total count query is running) you have a race condition which you can check the last_updated value of your full table scan count(). If it's > 24 hours old, then your incremental count query needs to pull a full day's count plus time elapsed today. If it's < 24 hours old, then your incremental count query needs to pull a partial day's count (just time elapsed today).