Firebase data to Google BigQuery

Question

Firebase offers private backups on Google Cloud Storage. One of the featured use case is \"Ingestion into Analytics Products\":

Private Backups provides a pe         


        

Answer 1

When writing this 03/2017, I can confirm that there's no real integration between Firebase Realtime database and BigQuery. Only Firebase Analytics can be imported easily into BigQuery. All this is not clearly explained on Firebase either...

We ended up writing our own solution, but you can check out this Github repo that has some 400+ stars, so I am assuming a few people found it useful...

Answer 2

In fact Big Query only support newline-delimited JSON or JSONL: https://cloud.google.com/bigquery/preparing-data-for-bigquery

http://jsonlines.org/

JSON Lines is a convenient format for storing structured data that may be processed one record at a time.

To prepare Firebase data for import in Big Query, we just need to:

1. Get the JSON from Firebase (or from a Cloud Storage bucket in case of private backup)
2. Parse it to get a JS object
3. Loop through each record, stringify data and add a line separator

       var dataForBigQuery = '';
       for (var i in dinosaurs) {
         dataForBigQuery+= JSON.stringify(dinosaurs[i]) + '\n';
       }
   

4. Save those data in a new file. It will then be ready for import into BigQuery.

Answer 3

Since the data above is just JSON, you should be able to get it to work with Firebase. However, I think that it would be much easier to prepare the data after the backup.

You mentioned that there was no arrays in the Firebase data. Firebase does support arrays, but they have to meet a certain criteria.

// we send this
['a', 'b', 'c', 'd', 'e']
// Firebase stores this
{0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e'}
// since the keys are numeric and sequential,
// if we query the data, we get this
['a', 'b', 'c', 'd', 'e']

Even though it may look like an object in the Firebase database, it will come back as an array when queried.

So it is feasible to create your schema in your Firebase database, but it would likely create a lot of overhead for your application.