How to unpack sqlite3 database written by Google AppEngine bulk downloader

Question

I need to download all instances of fairly large (multi-GB) entity in my app\'s datastore. I have enough disk space to store the entity\'s data, but not enough to store both

Answer 1

Entities are stored in the downloaded SQLite database as encoded Protocol Buffers (the same as they're stored in the production environment, and everywhere else - an entity is an encoded PB, in short). You can read them out yourself by using the SDK code for decoding entities (db.proto_to_entity() etc), but it'll be a bit of work to set everything up.

The relevant code is the ResultDatabase class in bulkloader.py - which you can probably reuse, along with other parts of the bulkloader, to make your job easier.