How to apply softmax on an array/vector with huge positive and negative values in TensorFlow?

问题

I train a convolutional neural network (CNN) with MNIST data set in TensorFlow. I calculate the accuracy for each image from the MNIST test images and looking for the values of the ten output-nodes. I use the following line of code to get it (see all code here: How to get the value from each output-node during eval MNIST testdata in TensorFlow?):

pred=prediction.eval(feed_dict={ x: testSet[0], y: testSet[1]})

The output of this line of code is for example this:

[[ -13423.92773438 -27312.79296875 20629.26367188 42987.953125
-34635.8203125 3714.84619141 -60946.6328125 106193.8828125
-20936.08789062 3940.52636719]]

When I try to apply the tf.nn.softmax() function at this vector/array with the following code:

pred_softmax = tf.nn.softmax(pred)
print(pred_softmax_new.eval())

I get for example this output:

[[ 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]]

But I am looking for results like this:

[[ 0.001 0.207 0.001 0.001 0.002 0.001 0.001 0.9723 0.001 0.001]]

I test this code:

test_a = tf.nn.softmax([31.0,23.0])
print(test_a.eval())

get this output:

[ 9.99664664e-01 3.35350138e-04]

But if I increase the value like:

test_a = tf.nn.softmax([45631.0,65423.0])

I get this output:

[ 0. 1.]

So my question is there a way to get good readable outputs for the ten output-nodes like for example this:

[[ 0.001 0.207 0.001 0.001 0.002 0.001 0.001 0.9723 0.001 0.001]]

回答1:

This seems like an issue of numerical stability mentioned here. I don't think that the answer mentioned in the link will help in your case though, as the differences between your numbers are extremely large. Basically, it seems that your model predict the value of 7 with probability 0.999999999....

来源：https://stackoverflow.com/questions/44893846/how-to-apply-softmax-on-an-array-vector-with-huge-positive-and-negative-values-i