What is the difference between np.mean and tf.reduce_mean?

Question

In the MNIST beginner tutorial, there is the statement

accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))

tf.cast

Answer 1

The key here is the word reduce, a concept from functional programming, which makes it possible for reduce_mean in TensorFlow to keep a running average of the results of computations from a batch of inputs.

If you are not familiar with functional programming, this can seem mysterious. So first let us see what reduce does. If you were given a list like [1,2,5,4] and were told to compute the mean, that is easy - just pass the whole array to np.mean and you get the mean. However what if you had to compute the mean of a stream of numbers? In that case, you would have to first assemble the array by reading from the stream and then call np.mean on the resulting array - you would have to write some more code.

An alternative is to use the reduce paradigm. As an example, look at how we can use reduce in python to calculate the sum of numbers: reduce(lambda x,y: x+y, [1,2,5,4]).

It works like this:

1. Step 1: Read 2 digits from the list - 1,2. Evaluate lambda 1,2. reduce stores the result 3. Note - this is the only step where 2 digits are read off the list
2. Step 2: Read the next digit from the list - 5. Evaluate lambda 5, 3 (3 being the result from step 1, that reduce stored). reduce stores the result 8.
3. Step 3: Read the next digit from the list - 4. Evaluate lambda 8,4 (8 being the result of step 2, that reduce stored). reduce stores the result 12
4. Step 4: Read the next digit from the list - there are none, so return the stored result of 12.

Read more here Functional Programming in Python

To see how this applies to TensorFlow, look at the following block of code, which defines a simple graph, that takes in a float and computes the mean. The input to the graph however is not a single float but an array of floats. The reduce_mean computes the mean value over all those floats.

import tensorflow as tf


inp = tf.placeholder(tf.float32)
mean = tf.reduce_mean(inp)

x = [1,2,3,4,5]

with tf.Session() as sess:
    print(mean.eval(feed_dict={inp : x}))

This pattern comes in handy when computing values over batches of images. Look at The Deep MNIST Example where you see code like:

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))