tensorflow-datasets

问题 I'm new to tensorflow keras and dataset. Can anyone help me understand why the following code doesn't work? import tensorflow as tf import tensorflow.keras as keras import numpy as np from tensorflow.python.data.ops import dataset_ops from tensorflow.python.data.ops import iterator_ops from tensorflow.python.keras.utils import multi_gpu_model from tensorflow.python.keras import backend as K data = np.random.random((1000,32)) labels = np.random.random((1000,10)) dataset = tf.data.Dataset.from

问题 I'm trying to use the TensorFlow Dataset API to read an HDF5 file, using the from_generator method. Everything works fine unless the batch size does not evenly divide into the number of events. I don't quite see how to make a flexible batch using the API. If things don't divide evenly, you get errors like: 2018-08-31 13:47:34.274303: W tensorflow/core/framework/op_kernel.cc:1263] Invalid argument: ValueError: `generator` yielded an element of shape (1, 28, 28, 1) where an element of shape (11

问题 Like pandas_df.shape is there any way for tensorflow.data.Dataset? Thanks. 回答1: I'm not familiar with something built-in, but the shapes could be retrieved from Dataset._tensors attribute. Example: import tensorflow as tf def dataset_shapes(dataset): try: return [x.get_shape().as_list() for x in dataset._tensors] except TypeError: return dataset._tensors.get_shape().as_list() And usage: from sklearn.datasets import make_blobs x_train, y_train = make_blobs(n_samples=10, n_features=2, centers=[

问题 I'm using a SequenceExample protobuf to read/write time-series data into a TFRecord file. I serialized a pair the np arrays as follows: writer = tf.python_io.TFRecordWriter(file_name) context = tf.train.Features( ... Feature( ... ) ... ) feature_data = tf.train.FeatureList(feature=[ tf.train.Feature(float_list=tf.train.FloatList(value= np.random.normal(size=([4065000,]))]) labels = tf.train.FeatureList(feature=[ tf.train.Feature(int64_list=tf.train.Int64List(value= np.random.random_integers(0

问题 I have a pre-trained model, and I'm trying to build another model that takes as input the output of the previous model. I don't want to train the models end-to-end, and want to use the first model for inference only. The first model was trained using tf.data.Dataset pipeline, and my first inclination was to integrate the model as just another dataset.map() operation at the tail of the pipeline, but I'm having issues with that. I've encountered 20 different errors in the process, each

问题 While adding the .cache() step to my dataset pipeline, successives training epochs still download the data from the network storage. I have a dataset on a network storage. I want to cache it, but not to repeat it: a training epoch must run through the whole dataset. Here is my dataset building pipeline: return tf.data.Dataset.list_files( file_pattern ).interleave( tf.data.TFRecordDataset, num_parallel_calls=tf.data.experimental.AUTOTUNE ).shuffle( buffer_size=2048 ).batch( batch_size=2048,

问题 What is the input size of faster RCNN RPN? I'm using an object detection API of Tensorflow which is using faster RCNN as region proposal network ( RPN ) and Inception as feature extractor ( according to the config file ). The API is using the online approach in prediction phase and detects every input image singly. however, I'm now trying to feed images to the network in the batch manner by use of Tensorflow dataset API. as you know for make batch out of the data, firstly we need to resize

问题 From your experience, what would be an ideal size of a .tfrecord file that would work best across a wide variety of devices (hard-disk, ssd, nvme) and storage locations (local machine, hpc cluster with network mounts) ? In case I get slower performance on a technically more powerful computer in the cloud than on my local PC, could the size of the tfrecord dataset be the root cause of the bottleneck ? Thanks 回答1: Official Tensorflow website recommends ~100MB (https://www.tensorflow.org

问题 My script used dataset API to realize input pipeline. When I use tensorflow 1.10, everything is ok, but when I upgrade to tensorflow 1.11 or 1.12. I got the following error message at the beginning of training: Attempted create an iterator on device "/job:localhost/replica:0/task:0/device:GPU:0" from handle defined on device "/job:localhost/replica:0/task:0/device:CPU:0" Here are piece of my code: def build_all_dataset(self): self.build_dataset(ROUTE_TRAIN) self.build_dataset(ROUTE_VALIDATION

问题 I am trying tensorflow course from Udacity which uses google colab to write/run the code. But I want to run the code on my local machine and hence have created a new environment to run the code , but am unable to import tensorflow_dataset into the tensorflow environment . I have tried searching for the module from the anaconda navigator to install it in tensorflow environment but anaconda navigator does not provide the tensorflow datasets module. From cmd (command prompt) I tried to execute