External Backups/Snapshots for Google Cloud Spanner

Question

Is it possible to snapshot a Google Cloud Spanner Database/table(s)? For compliance reasons we have to have daily snapshots of the current database that can be rolled back to in

Answer 1

As of July 2018, Cloud Spanner now offers Import and Export functionality which allows you to export a database into Avro format. If you go to the specific Cloud Spanner database in question via the Google Cloud Console website, you will see Import and Export buttons toward the top. Simply click Export, populate the requested information such as a destination Google Cloud Storage bucket, and the database will be backed-up in Avro format to Google Cloud Storage. If you need to restore a database, use the corresponding Import functionality from the Cloud Spanner section of the Google Cloud Console website.

Note: The actual backup and restore (i.e., export and import) are done using Google Cloud Dataflow and you will be charged for the dataflow operation.

See the documentation for Import and Export functionality.

Answer 2

Today, you can stream out a consistent snapshot by reading out all the data using your favorite tool (mapreduce, spark, dataflow) and reads at a specific timestamp (using Timestamp Bounds).

https://cloud.google.com/spanner/docs/timestamp-bounds

You have about an hour to do the export before the data gets garbage collected.

In the future, we will provide a Apache Beam/Dataflow connector to do this in a more scalable fashion. This will be our preferred method for doing import/export of data into Cloud Spanner.

Longer term, we will support backups and the ability to restore to a backup but that functionality is not currently available.

Answer 3

Google Cloud Spanner now has two methods that can be used to do backups.

https://cloud.google.com/spanner/docs/backup

You can either use the built-in backups or do an export/import using a dataflow job.

Answer 4

Per the thread below (also answered by eb80), I have been able to successfully script my backups from Spanner using their Import/Export DataFlow jobs by building the templates and running the gcloud command out of a cron job. The catch is that when you schedule the job you need to parse the output and get the job ID so that you can then check the status of it (again, I scripted that) and when you get JOB_STATE_DONE or JOB_STATE_FAILED, it's complete. If you need to run the Import, you'll need that job ID because the specific backup folder structure is:

gs://bucketName/yourFolder/instanceId-databaseId-jobId

How to batch load custom Avro data generated from another source?