Handling Schema Migrations in App Engine

Question

I\'ve updated several of the heavily-used NDB models in one of my App Engine applications by moving a few properties between them. As such, some models now contain data that pre

Answer 1

You could do 2 migrations (add + delete) instead of a single (move) one.

A first migration just adds the properties to be moved into their new locations in the models.

Update the code reading the properties to check the new locations first and, if the properties don't exist, fallback to the old locations. Queries will likely need to be doubled for the 2 locations and with logic added to identify and skip the duplicate results. Pretty much a combination of the new and the old code.

Then you can update the code writing the properties to use the new locations. Entities created past this moment won't have the old locations anymore. If you want to play it even safer (able to rollback to older code versions) or if you want to keep your older app versions running for a little while longer you can make the writing code write in both old and new locations.

Then run a one-time job copying the attribute contents from the old locations (if they exist) to the new locations. This ensures all properties are present in the new locations for all entities.

At this moment you can drop all code accessing the old locations.

This is the point of no return. Before the next step you need to stop any GAE instance running your older app code - they won't run properly anymore and you won't be able to rollback your code without another DB-updating job.

Then run a one-time cleanup job removing the properties from the old locations for all entities - they're no longer accessed by the lastest version of the code.

Finalize with a 2nd migration deleting the properties from the old location in the models.

It might be a lot of work, but it should be possible to achieve zero downtime (for the latest app version) irrespective of the duration of the migrations.

Update:

A potentially even better migration algorithm (avoids problematic OR'd doubled queries) is described in How to rename a Datastore entity field but be able to retrieve records via old and new property names?