Using binary_crossentropy loss in Keras (Tensorflow backend)

Question

In the training example in Keras documentation,

https://keras.io/getting-started/sequential-model-guide/#training

binary_crossentropy is use

Answer 1

In Keras by default we use activation sigmoid on the output layer and then use the keras binary_crossentropy loss function, independent of the backend implementation (Theano, Tensorflow or CNTK).

If you look more in depth for the pure Tensorflow case you find that the tensorflow backend binary_crossentropy function (which you pasted in your question) uses tf.nn.sigmoid_cross_entropy_with_logits. The later function also add the sigmoid activation. To avoid double sigmoid, the tensorflow backend binary_crossentropy, will by default (with from_logits=False) calculate the inverse sigmoid (logit(x)=log(x/1-x)) to get the output back into the raw state from the network with no activation.

The extra activation sigmoid, and inverse sigmoid calculation can be avoided by using no sigmoid activation function in your last layer, and then call the tensorflow backend binary_crossentropy with parameter from_logits=True (Or directly use tf.nn.sigmoid_cross_entropy_with_logits)

Answer 2

You're right, that's exactly what's happening. I believe this is due to historical reasons.

Keras was created before tensorflow, as a wrapper around theano. And in theano, one has to compute sigmoid/softmax manually and then apply cross-entropy loss function. Tensorflow does everything in one fused op, but the API with sigmoid/softmax layer was already adopted by the community.

If you want to avoid unnecessary logit <-> probability conversions, call binary_crossentropy loss withfrom_logits=True and don't add the sigmoid layer.

Answer 3

In categorical cross entropy :

• if it is prediction it will compute the cross entropy directly
• if it is logit it will apply softmax_cross entropy with logit

In Binary cross entropy:

• if it is prediction it will convert it back to logit then apply sigmoied cross entropy with logit
• if it is logit it will apply sigmoied cross entropy with logitdirectly