Google Cloud ML and GCS Bucket issues

Question

I\'m using open source Tensorflow implementations of research papers, for example DCGAN-tensorflow. Most of the libraries I\'m using are configured to train the model locall

Answer 1

There is currently no definitive guide. The basic idea would be to replace all occurrences of native Python file operations with equivalents in the file_io module, most notably:

• open() -> file_io.FileIO()
• os.path.exists() -> file_io.file_exists()
• glob.glob() -> file_io.get_matching_files()

These functions will work locally and on GCS (as well as any registered file system). Note, however, that there are some slight differences in file_io and the standard file operations (e.g., a different set of 'modes' are supported).

Fortunately, checkpoint and summary writing do work out of the box, just be sure to pass a GCS path to tf.train.Saver.save and tf.summary.FileWriter.

In the sample you sent, that looks potentially painful. Consider monkey patching the Python functions to map to the TensorFlow equivalents when the program starts to only have to do it once (demonstrated here).

As a side note, all of the samples on this page show reading files from GCS.