Why does binary accuracy give high accuracy while categorical accuracy give low accuracy, in a multi-class classification problem?

Question

I\'m working on a multiclass classification problem using Keras and I\'m using binary accuracy and categorical accuracy as metrics. When I evaluate my model I get a really high

Answer 1

So you need to understand what happens when you apply a binary_crossentropy to a multiclass prediction.

1. Let's assume that your output from softmax is (0.1, 0.2, 0.3, 0.4) and one-hot encoded ground truth is (1, 0, 0, 0).
2. binary_crossentropy masks all outputs which are higher than 0.5 so out of your network is turned to (0, 0, 0, 0) vector.
3. (0, 0, 0, 0) matches ground truth (1, 0, 0, 0) on 3 out of 4 indexes - this makes resulting accuracy to be at the level of 75% for a completely wrong answer!

To solve this you could use a single class accuracy, e.g. like this one:

def single_class_accuracy(interesting_class_id):
    def fn(y_true, y_pred):
        class_id_preds = K.argmax(y_pred, axis=-1)
        # Replace class_id_preds with class_id_true for recall here
        positive_mask = K.cast(K.equal(class_id_preds, interesting_class_id), 'int32')
        true_mask = K.cast(K.equal(y_true, interesting_class_id), 'int32')
        acc_mask = K.cast(K.equal(positive_mask, true_mask), 'float32')
        class_acc = K.mean(acc_mask)
        return class_acc

    return fn