Specificity in scikit learn

Question

I need specificity for my classification which is defined as : TN/(TN+FP)

I am writing a custom scorer function :

from sklearn.         


        

Answer 1

I appreciate this is an old question but thought I would mention sklearn pretty much does this (at least in scikit-learn v0.21.2, but I'm confident it's been around for ages)

As I understand it, 'specificity' is just a special case of 'recall'. Recall is calculated for the actual positive class ( TP / [TP+FN] ), whereas 'specificity' is the same type of calculation but for the actual negative class ( TN / [TN+FP] ).

It really only makes sense to have such specific terminology for binary classification problems. For a multi-class classification problem it would be more convenient to talk about recall with respect to each class. There is no reason why you can't talk about recall in this way even when dealing with binary classification problem (e.g. recall for class 0, recall for class 1).

For example, recall tells us the proportion of patients that actual have cancer, being successfully diagnosed as having cancer. However, to generalize, you could say Class X recall tells us the proportion of samples actually belonging to Class X, being successfully predicted as belonging to Class X.

Given this, you can use from sklearn.metrics import classification_report to produce a dictionary of the precision, recall, f1-score and support for each label/class. You can also rely on from sklearn.metrics import precision_recall_fscore_support as well, depending on your preference. Documentation here.

Answer 2

First of all you need to know that:

DummyClassifier(strategy='most_frequent'...

Will give you classifier which returns most frequent label from your training set. It doesn't even take into consideration samples in X. You can pass anything instead of ground_truth in this line:

clf_dummy = clf_dummy.fit(ground_truth, p)

result of training, and predictions will stay same, because majority of labels inside p is label "0".

Second thing that you need to know: make_scorer returns function with interface scorer(estimator, X, y) This function will call predict method of estimator on set X, and calculates your specificity function between predicted labels and y.

So it calls clf_dummy on any dataset (doesn't matter which one, it will always return 0), and returns vector of 0's, then it computes specificity loss between ground_truth and predictions. Your predictions is 0 because 0 was majority class in training set. Your score is equals 1 because there is no false positive predictions.

I corrected your code, to add more convenience.

from sklearn.dummy import DummyClassifier
clf_dummy = DummyClassifier(strategy='most_frequent', random_state=0)
X = [[0],[0],[1],[0],[1],[1],[1],[0],[0],[1],[0],[0],[1]]
p  = [0,0,0,1,0,1,1,1,1,0,0,1,0]
clf_dummy = clf_dummy.fit(X, p)
score(clf_dummy, X, p)

Answer 3

Remembering that in binary classification, recall of the positive class is also known as “sensitivity”; recall of the negative class is “specificity”, I use this:

unique, counts = np.unique(y_test, return_counts=True)

for i in unique:
    score = precision_score(y_true, y_pred, labels=unique, pos_label=i)
    print('score ' + str(i) + '  ' + str(score))

Answer 4

You could get specificity from the confusion matrix. For a binary classification problem, it would be something like:

from sklearn.metrics import confusion_matrix
y_true = [0, 0, 0, 1, 1, 1, 1, 1]
y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
specificity = tn / (tn+fp)