Load Google Cloud Storage data into bigtable

Question

Is there an easy way or example to load Google Cloud Storage data into bigtable?

I have lots of json files generated by pyspar

Answer 1

There is no simple tool to read data in Cloud Bigtable. Here are some options:

1. Import the files using Dataflow. This requires java development, and learning the Dataflow programming model.
2. Use Python (possibly with Pyspark) to read those json files, and write to Cloud Bigtable using a method called mutate_rows which write to Bigtable in bulk.

FYI, I work on the Cloud Bigtable team. I'm a Java developer, so I opt for #1. Our team has been working to improve our python experience. The extended team recently added some reliability improvements to make sure that mutate_rows is resilient for large jobs. We do not yet have any good examples of integrating with PySpark or Apache Beam's python SDK, but they are on our radar.