Android SQLite and huge data sets

Question

We are creating an app for a client that has hundreds of megabytes of HTML in SQLite databases. We have implemented a way to query this data and scroll through it all in a r

Answer 1

In my experience, limiting queries makes it take a lot longer to get results because starting new cursors is expensive for low limits. I tried doing 62k rows with 1k and even 10k limit just now and it is very slow and unusable since I have to start more than 6 cursors. I'll just stick with not supporting 2.3.3.... It is the latest build that I get CursorWindow ERROR instead of WARN.

Here is what I did though. This is probably the best algorithm I think. Don't have to do queries twice etc. However, it can get pretty slow with big queries and small limits so you have to test out what works best a lot. In my case, it's not fast enough for my purposes since it doesn't handle 62k rows well.

int cursorCount = 0;
int limit = 1000; //whatever you want
while (true)
{
    Cursor cursor = builder.query(mDatabaseHelper.getReadableDatabase(), columns, selection, selectionArgs, null, null, null, Integer.toString(limit));
    if (cursor == null) {
        return null;
    } else if (!cursor.moveToFirst()) { //if it is empty, return null
        return null;
    }
    cursorCount = cursor.getCount();
    if (cursorCount % limit != 0)
    {
        return cursor;
    }
    limit+=1000; //same amount as the one above
} 

Answer 2

If you really need that you can also split your data and read chunks like this:

   int limit = 0;
   while (limit + 100 < numberOfRows) {
       //Compose the statement
       String statement = "SELECT * FROM Table ORDER someField LIMIT '"+ limit+"', 100";
       //Execute the query
       Cursor cursor = myDataBase.rawQuery(statement, null);
       while (cursor.moveToNext()) {
           Product product = new Product();
           product.setAllValuesFromCursor(cursor);
           productsArrayList.add(product);
      }
      cursor.close();
      limit += 100;
 }

 //Compose the statement
 String statement = "SELECT * FROM Table ORDER someField LIMIT '"+  (numberOfRows - limit)+"', 100";
 //Execute the query
 Cursor cursor = myDataBase.rawQuery(statement, null);

 while (cursor.moveToNext()) {
     Product product = new Product();
     product.setAllValuesFromCursor(cursor);
     productsArrayList.add(product);
 }
 cursor.close();

It's working under 2 s for 5k rows if you have indexed table.

Thanks, Arkde

Answer 3

what options do we have in querying and displaying tens of thousands of rows of data in Android?

You mean besides telling you that reading 20,000+ rows on a 3.5" LCD is bat-guano crazy? ;-)

It looks like CursorWindow, which is used somewhere under the covers, is having issues managing >17,000 rows. That could be one of two things:

1. You are out of heap space. With a 16MB non-compacting heap, and the fact that a Cursor holds the entire result set in the heap, that is not out of the question.
2. CursorWindow only supports 1MB of data, which is what the error message suggests more directly.

If there is a logical way to divide your queries into discrete chunks, you could do incremental queries and use CursorJoiner to stitch them together, and see if that helps.

But, in all seriousness, 20,000+ rows in a 3.5" screen, on a device that most closely resembles a 12-year-old PC in horsepower, is really asking a lot.