How do I read the contents of a new cloud storage file of type .json from within a cloud function?

Question

The event passed to my Google cloud function only really tells me the name of the bucket and file, and whether the file was deleted. Yes, there\'s more there, but i

Answer 1

You need to use a client library for google storage instead of accessing via the URL. Using request() against the URL would only work if the file was exposed to public access.

Import the google cloud storage library in the npm-managed directory containing your project.

npm i @google-cloud/storage -S

The npm page for google-cloud/storage has decent examples but I had to read through the API a bit to see an easy way to download to memory.

Within the Google Cloud Functions environment, you do not need to supply any api key, etc. to storage as initialization.

const storage = require('@google-cloud/storage')();

The metadata passed about the file can be used to determine if you really want the file or not.

When you want the file, you can download it with the file.download function, which can take either a callback or, lacking a callback, will return a promise.
The data however, is returned as a Buffer so you will need to call data.toString('utf-8') to convert it to a utf-8 encoded string.

const storage = require('@google-cloud/storage')();

exports.logNewJSONFiles = function logNewJSONFiles(event){
    return new Promise(function(resolve, reject){
        const file = event.data;
        if (!file){
            console.log("not a file event");
            return resolve();
        }
        if (file.resourceState === 'not_exists'){
            console.log("file deletion event");
            return resolve();
        }
        if (file.contentType !== 'application/json'){
            console.log("not a json file");
            return resolve();
        }
        if (!file.bucket){
            console.log("bucket not provided");
            return resolve();
        }
        if (!file.name){
            console.log("file name not provided");
            return resolve();
        }
        (storage
         .bucket(file.bucket)
         .file(file.name)
         .download()
         .then(function(data){
             if (data)
                 return data.toString('utf-8');
         })
         .then(function(data){
             if (data) {
                 console.log("new file "+file.name);
                 console.log(data);
                 resolve(data);
             }
         })
         .catch(function(e){ reject(e); })
             );
    });
};

Deployment is as expected:

gcloud beta functions deploy logNewJSONFiles --stage-bucket gs://my-stage-bucket --trigger-bucket gs://my-echo-bucket

Remember to look in the Stackdriver:Logging page on Google Cloud Platform for the console.log entries.

UPDATE: (2019) When cloud-functions first released, they had some issues with ECONNRESET. I think that's fixed now. If not, use something like npm:promise-retry