google-cloud-datalab

loading saved keras model from gs to pydatalab

隐身守侯 提交于 2019-12-07 23:59:44
问题 My keras model is saved in google storage with model.save(model_name) I cannot load the model on pydatalab. When I save the model on my local machine, I can just open it with load_model(filepath). Also I did import keras.backend as K, based on NameError when opening Keras model that uses Tensorflow Backend I have tried the following: model = load_model(tf.gfile.Open(model_file)) Error: TypeError: expected str, bytes or os.PathLike object, not GFile load_model('gs://mybucket/model.h5') Error:

Google Datalab) How to read multiple csv files existing in Google Cloud Storage by pandas read_csv() to?

纵然是瞬间 提交于 2019-12-07 21:01:33
问题 I found the solution for reading a "single" csv file in a Datalab : How can i load my csv from google dataLab to a pandas data frame? But I wonder how I could read "multiple" csv files in Datalab . What I tried is like this : variable_list = ['IBM', 'SPY'] for variable in variable_list: file_path = "gs://chois-trader-bucket/data/" + variable + ".csv" %%storage read --object file_path --variable variable But this one failed because python variable is not compatible with magic command. How can

Reading batches of data from BigQuery into Datalab

∥☆過路亽.° 提交于 2019-12-07 16:48:47
问题 I have a big dataset in BigQuery table (~45M lines, 13Gb of data). I would like to process that data in my Google Datalab Notebook to do some basic statistics with pandas to visualise data later with matplotlib in Datalab cell. I think it is not a good idea to try to load all dataset into pandas' Dataframe (at least I will have RAM issues). Is it possible to read data from BigQuery in batches (say 10K lines) to consume it in Datalab? Thanks in advance! 回答1: If your purpose is to visualize the

Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

混江龙づ霸主 提交于 2019-12-07 12:08:26
问题 I have been trying to write a function that loads multiple files from a Google Cloud Storage bucket into a single Pandas Dataframe, however I cannot seem to make it work. import pandas as pd from google.datalab import storage from io import BytesIO def gcs_loader(bucket_name, prefix): bucket = storage.Bucket(bucket_name) df = pd.DataFrame() for shard in bucket.objects(prefix=prefix): fp = shard.uri %gcs read -o $fp -v tmp df.append(read_csv(BytesIO(tmp)) return df When I try to run it says:

launching tensorboard from google cloud datalab

ε祈祈猫儿з 提交于 2019-12-07 03:53:58
问题 I need help in luanching tensorboard from tensorflow running on the datalab, My code is the followings (everything is on the datalab): import tensorflow as tf with tf.name_scope('input'): print ("X_np") X_np = tf.placeholder(tf.float32, shape=[None, num_of_features],name="input") with tf.name_scope('weights'): print ("W is for weights & - 15 number of diseases") W = tf.Variable(tf.zeros([num_of_features,15]),name="W") with tf.name_scope('biases'): print ("b") #todo:authemate for more diseases

Reading in a file with Google datalab

爷,独闯天下 提交于 2019-12-07 03:14:46
问题 I am trying to use Google Datalab to read in a file in ipython notebook, the basic pd.read_csv() as I can't find the path of the file. I have it locally and also uploaded it to google cloud storage in a bucket. I ran the following commands to understand where I am os.getcwd() gives '/content/myemail@gmail.com' os.listdir('/content/myemail@gmail.com') gives ['.git', '.gitignore', 'datalab', 'Hello World.ipynb', '.ipynb_checkpoints'] 回答1: The following reads the contents of the object into a

Google Cloud Datalab error writing to Cloud Storage

情到浓时终转凉″ 提交于 2019-12-06 14:47:57
I am using Google Cloud Datalab for the first time to build a classifier for a Kaggle competition. But I am stuck trying to write a csv file containing the pre-processed training data to Cloud Storage using the google.datalab.storage API. The file contains strings with unicode characters which causes the write_stream to a Storage object to trigger the error: Failed to process HTTP response. Here is the simplified code only trying to write a single string: from google.datalab import Context import google.datalab.storage as storage project = Context.default().project_id bucket_name = project

Google Datalab) How to read multiple csv files existing in Google Cloud Storage by pandas read_csv() to?

…衆ロ難τιáo~ 提交于 2019-12-06 14:01:11
I found the solution for reading a "single" csv file in a Datalab : How can i load my csv from google dataLab to a pandas data frame? But I wonder how I could read "multiple" csv files in Datalab . What I tried is like this : variable_list = ['IBM', 'SPY'] for variable in variable_list: file_path = "gs://chois-trader-bucket/data/" + variable + ".csv" %%storage read --object file_path --variable variable But this one failed because python variable is not compatible with magic command. How can I deal with multiple csv files effectively? You can use variables enclosed in braces. e.g) %storage

How to execute a python notebook inside another one at google cloud datalab

删除回忆录丶 提交于 2019-12-06 13:38:46
I'd like to execute a python notebook I'had created to data pre-processing inside another notebook related with data classification process. So the last notebook depends on the functions and execution provided by the first notebook. How could I do that at google cloud datalab environment? I do like to reuse the functions and variables used at the pre-processing notebook on the classification notebook. Thanks. The following should work: myNotebook = <relative path to notebook> %run $myNotebook or simply %run <relative path to notebook> Note: This does not work if you are running Datalab on

How to add 'private' python module to Google Datalab

断了今生、忘了曾经 提交于 2019-12-06 07:18:32
问题 I'm experimenting with the promising Google Cloud Datalab. In the past I've created some handy python classes and functions that I'd like to use in the GCD-notebooks but I don't know how to add my code. Anybody any suggestions? 回答1: This is, on second thought, obvious: %%bash pip install git+http://myawsomepythonmodule.git 来源: https://stackoverflow.com/questions/33165443/how-to-add-private-python-module-to-google-datalab