How to average summaries over multiple batches?

Question

Assuming I have a bunch of summaries defined like:

loss = ...
tf.scalar_summary(\"loss\", loss)
# ...
summaries = tf.m         


        

Answer 1

Do the averaging of your measure in Python and create a new Summary object for each mean. Here is what I do:

accuracies = []

# Calculate your measure over as many batches as you need
for batch in validation_set:
  accuracies.append(sess.run([training_op]))

# Take the mean of you measure
accuracy = np.mean(accuracies)

# Create a new Summary object with your measure
summary = tf.Summary()
summary.value.add(tag="%sAccuracy" % prefix, simple_value=accuracy)

# Add it to the Tensorboard summary writer
# Make sure to specify a step parameter to get nice graphs over time
summary_writer.add_summary(summary, global_step)

Answer 2

I would avoid calculating the average outside the graph.

You can use tf.train.ExponentialMovingAverage:

ema = tf.train.ExponentialMovingAverage(decay=my_decay_value, zero_debias=True)
maintain_ema_op = ema.apply(your_losses_list)

# Create an op that will update the moving averages after each training step.
with tf.control_dependencies([your_original_train_op]):
    train_op = tf.group(maintain_ema_op)

Then, use:

sess.run(train_op)

That will call maintain_ema_op because it is defined as a control dependency.

In order to get your exponential moving averages, use:

moving_average = ema.average(an_item_from_your_losses_list_above)

And retrieve its value using:

value = sess.run(moving_average)

This calculates the moving average within your calculation graph.

Answer 3

You can average store the current sum and recalculate the average after each batch, like:

loss_sum = tf.Variable(0.)
inc_op = tf.assign_add(loss_sum, loss)
clear_op = tf.assign(loss_sum, 0.)
average = loss_sum / batches
tf.scalar_summary("average_loss", average)

sess.run(clear_op)
for i in range(batches):
    sess.run([loss, inc_op])

sess.run(average)

Answer 4

As of August 2018, streaming metrics have been depreciated. However, unintuitively, all metrics are streaming. So, use tf.metrics.accuracy.

However, if you want accuracy (or another metric) over only a subset of batches, then you can use Exponential Moving Average, as in the answer by @MZHm or reset any of the the tf.metric's by following this very informative blog post

Answer 5

I think it's always better to let tensorflow do the calculations.

Have a look at the streaming metrics. They have an update function to feed the information of your current batch and a function to get the averaged summary. It's going to look somewhat like this:

accuracy = ... 
streaming_accuracy, streaming_accuracy_update = tf.contrib.metrics.streaming_mean(accuracy)
streaming_accuracy_scalar = tf.summary.scalar('streaming_accuracy', streaming_accuracy)

# set up your session etc. 

for i in iterations:
      for b in batches:
               sess.run([streaming_accuracy_update], feed_dict={...})

     streaming_summ = sess.run(streaming_accuracy_scalar)
     writer.add_summary(streaming_summary, i)

Also see the tensorflow documentation: https://www.tensorflow.org/versions/master/api_guides/python/contrib.metrics

and this question: How to accumulate summary statistics in tensorflow

Answer 6

For future reference, the TensorFlow metrics API now supports this by default. For example, take a look at tf.mean_squared_error:

For estimation of the metric over a stream of data, the function creates an update_op operation that updates these variables and returns the mean_squared_error. Internally, a squared_error operation computes the element-wise square of the difference between predictions and labels. Then update_op increments total with the reduced sum of the product of weights and squared_error, and it increments count with the reduced sum of weights.

These total and count variables are added to the set of metric variables, so in practice what you would do is something like:

x_batch = tf.placeholder(...)
y_batch = tf.placeholder(...)
model_output = ...
mse, mse_update = tf.metrics.mean_squared_error(y_batch, model_output)
# This operation resets the metric internal variables to zero
metrics_init = tf.variables_initializer(
    tf.get_default_graph().get_collection(tf.GraphKeys.METRIC_VARIABLES))
with tf.Session() as sess:
    # Train...
    # On evaluation step
    sess.run(metrics_init)
    for x_eval_batch, y_eval_batch in ...:
        mse = sess.run(mse_update, feed_dict={x_batch: x_eval_batch, y_batch: y_eval_batch})
    print('Evaluation MSE:', mse)