tfrecord | 易学教程

Decoding tfrecord with tfslim

阅读更多关于 Decoding tfrecord with tfslim

I use Python 2.7.13 and Tensorflow 1.3.0 on CPU. I want to use DensNet( https://github.com/pudae/tensorflow-densenet ) for regression problem. My data contains 60000 jpeg images with 37 float labels for each image. I saved my data into tfrecords files by: def Read_Labels(label_path): labels_csv = pd.read_csv(label_path) labels = np.array(labels_csv) return labels[:,1:] ` def load_image(addr): # read an image and resize to (224, 224) img = cv2.imread(addr) img = cv2.resize(img, (224, 224), interpolation=cv2.INTER_CUBIC) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) img = img.astype(np.float32)

Getting batch predictions for TFrecords via CloudML

阅读更多关于 Getting batch predictions for TFrecords via CloudML

问题 I followed this great tutorial and successfully trained a model (on CloudML). My code also makes predictions offline, but now I am trying to use Cloud ML to make predictions and have some problems. To deploy my model I followed this tutorial. Now I have a code that generates TFRecords via apache_beam.io.WriteToTFRecord and I want to make predictions for those TFRecords . To do so I am following this article, my command looks like this: gcloud ml-engine jobs submit prediction $JOB_ID --model

Obtaining total number of records from .tfrecords file in Tensorflow

阅读更多关于 Obtaining total number of records from .tfrecords file in Tensorflow

问题 Is it possible for obtain the total number of records from a .tfrecords file ? Related to this, how does one generally keep track of the number of epochs that have elapsed while training models? While it is possible for us to specify the batch_size and num_of_epochs , I am not sure if it is straightforward to obtain values such as current epoch , number of batches per epoch etc - just so that I could have more control of how the training is progressing. Currently, I'm just using a dirty hack

How to use Dataset API to read TFRecords file of lists of variant length?

阅读更多关于 How to use Dataset API to read TFRecords file of lists of variant length?

I want to use Tensorflow's Dataset API to read TFRecords file of lists of variant length. Here is my code. def _int64_feature(value): # value must be a numpy array. return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) def main1(): # Write an array to TFrecord. # a is an array which contains lists of variant length. a = np.array([[0, 54, 91, 153, 177], [0, 50, 89, 147, 196], [0, 38, 79, 157], [0, 49, 89, 147, 177], [0, 32, 73, 145]]) writer = tf.python_io.TFRecordWriter('file') for i in range(a.shape[0]): # i = 0 ~ 4 x_train = a[i] feature = {'i': _int64_feature(np.array([i])),

How to use Dataset API to read TFRecords file of lists of variant length?

阅读更多关于 How to use Dataset API to read TFRecords file of lists of variant length?

问题 I want to use Tensorflow's Dataset API to read TFRecords file of lists of variant length. Here is my code. def _int64_feature(value): # value must be a numpy array. return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) def main1(): # Write an array to TFrecord. # a is an array which contains lists of variant length. a = np.array([[0, 54, 91, 153, 177], [0, 50, 89, 147, 196], [0, 38, 79, 157], [0, 49, 89, 147, 177], [0, 32, 73, 145]]) writer = tf.python_io.TFRecordWriter('file')

TensorFlow - Read all examples from a TFRecords at once?

阅读更多关于 TensorFlow - Read all examples from a TFRecords at once?

问题 How do you read all examples from a TFRecords at once? I've been using tf.parse_single_example to read out individual examples using code similar to that given in the method read_and_decode in the example of the fully_connected_reader. However, I want to run the network against my entire validation dataset at once, and so would like to load them in their entirety instead. I'm not entirely sure, but the documentation seems to suggest I can use tf.parse_example instead of tf.parse_single

TensorFlow - Read video frames from TFRecords file

阅读更多关于 TensorFlow - Read video frames from TFRecords file

问题 TLDR; my question is on how to load compressed video frames from TFRecords. I am setting up a data pipeline for training deep learning models on a large video dataset (Kinetics). For this I am using TensorFlow, more specifically the tf.data.Dataset and TFRecordDataset structures. As the dataset contains ~300k videos of 10 seconds, there is a large amount of data to deal with. During training, I want to randomly sample 64 consecutive frames from a video, therefore fast random sampling is

Numpy to TFrecords: Is there a more simple way to handle batch inputs from tfrecords?

阅读更多关于 Numpy to TFrecords: Is there a more simple way to handle batch inputs from tfrecords?

My question is about how to get batch inputs from multiple (or sharded) tfrecords. I've read the example https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410 . The basic pipeline is, take the training set as as example, (1) first generate a series of tfrecords (e.g., train-000-of-005 , train-001-of-005 , ...), (2) from these filenames, generate a list and fed them into the tf.train.string_input_producer to get a queue, (3) simultaneously generate a tf.RandomShuffleQueue to do other stuff, (4) using tf.train.batch_join to generate batch inputs. I think