tensorflow: efficient feeding of eval/train data using queue runners

Question

I\'m trying to run a tensorflow graph to train a model and periodically evaluate using a separate evaluation dataset. Both training and evaluation data is implemented using

Answer 1

After some experimentation, my current best solution is to have a main graph featuring training inputs and a separate graph with just evaluation data operations. I open a separate session to get evaluation data and feed this to the training graph when I want to evaluate. Highly inelegant (and evaluation runs take longer than they ideally would as they have to come ot of one session only to be fed to another), but assuming evaluation runs are rare compared to training runs, this seems preferable to the original version...

import tensorflow as tf
from tensorflow.models.image.cifar10 import cifar10
from time import time


class DataSupplier:
    def __init__(self, tensor_fn):
        graph = tf.Graph()
        with graph.as_default():
            with graph.device('/cpu:0'):
                self.tensor = tensor_fn()
        self.sess = tf.Session(graph=graph)
        self.coord = tf.train.Coordinator()
        self.threads = tf.train.start_queue_runners(sess=self.sess,
                                                    coord=self.coord)

    def get_tensor_val(self):
        return self.sess.run(self.tensor)

    def clean_up(self):
        self.coord.request_stop()
        self.coord.join(self.threads)


eval_batcher = DataSupplier(lambda: cifar10.inputs(True))

graph = tf.Graph()
with graph.as_default():
    images, labels = cifar10.inputs(False)

    out_images = tf.identity(images)
    out_labels = tf.identity(labels)

n_runs = 100

with tf.Session(graph=graph) as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess, coord)
    for i in range(n_runs):
        sess.run([out_images, out_labels])
    t = time()
    for i in range(n_runs):
        sess.run([out_images, out_labels])
    dt = (time() - t)/n_runs
    print('Train time: %.3f' % dt)
    t = time()
    for i in range(n_runs):
        eval_images, eval_labels = eval_batcher.get_tensor_val()
        sess.run([out_images, out_labels],
                 feed_dict={images: eval_images, labels: eval_labels})
    dt = (time() - t)/n_runs
    print('Eval time: %.3f' % dt)
    coord.request_stop()
    coord.join(threads)

eval_batcher.clean_up()

Results:

Train time: 0.050
Eval time: 0.064

Update: when using this approach in training problems with tf.contrib.layers and regularization, I find the regularization losses go to infinity if the DataSupplier graph is on the same device as the training graph. I cannot for the life of me explain why this is the case, but explicitly setting the device of the DataSupplier to the CPU (given the training graph is on my GPU) seems to work...

Answer 2

Have you read the last section of this link about multi inputs? I think you can add a is_training argument to your input function to distinguish training data from eval data. Then you can reuse sharing variables to get the logits for eval data and build a op for eval. Then in your graph, run valudation_accuracy=sess.run(eval_op) to get eval accuracy.

---

Update:

Hi, from my understanding,if you want to train for n batches, evaluate, train, evaluate, you can keep there two ops in the same graph, no need to build a new one. Assume you have already build all the needed function, then the code should like this:

#the following two steps will add train and eval input queue to the graph
train_inputs,train_labels = inputs(is_train=True)
eval_inputs,eval_labels = inputs(is_train=False)

with tf.variable_scope("inference") as scope:
    train_logits = inference(train_inputs)
    scope.reuse_variables()
    eval_logits = inference(eval_inputs)

loss = loss(train_logits,train_labels)
eval_accuracy = accuracy(eval_logits,eval_labels)

#...add train op here,start queue runner and train it ...