AUC-ROC for a none ranking Classifier such as OSVM

Question

Im currently working with auc-roc curves , and lets say that I have a none ranking Classifier such as a one class SVM where the predictions are either 0 and 1 and the predic

Answer 1

The standard ROC curve requires varying the probability or score threshold of your classifier, and obtaining the corresponding graph of the ordered pairs of (true positive rate, false positive rate) for each varied threshold value.

Since the One-Class SVM is defined in such a way that it does not produce probability results or scores as part of its output (this is specifically different than standard SVM classifiers), it means that a ROC curve is inapplicable unless you create your own version of a score as discussed below.

Furthermore, the training for a One-Class SVM is specifically hugely imbalanced, because the training data is solely a set of "positive" examples, e.g. observations that come from the distribution in question. ROC curves would suffer greatly from large class imbalance anyway, so the ROC curve could be misleading in the sense that the classification score for a small number of outliers would be hugely more important than the score for a bunch of non-outliers at the heart of the observed distribution's highest density areas. So avoiding ROC for this type of model, even if you create your own scores, is advisable.

You are correct to choose precision vs. recall as a better metric, but in the plot you show in your question, you are still overlaying a plot on top of true positive rate and false positive rate along the axes, while the AUC-pr (precision recall AUC score) looks like it is just a single point padded with 0 for the false positive rate (e.g. it is purely a bug in your code for plotting).

In order to get an actual precision recall curve, you need some way of associating a score to the outlier decision. One suggestion is to use the decision_function attribute of the fitted OneClassSVM object after training.

If you compute the maximum of decision_function(x) over all input values x, call this MAX, then one way of associating a score is to treat the score for the prediction on some data y as score = MAX - decision_function(y).

This assumes you have the labels set up in such a way that large value of decision_function(x) means x is not an outlier, so it does have the label of the positive class used for training. You could take the reciprocal or use other transformations if you set up your problem with reverse labels (meaning, whether you set the OneClassSVM to predict '1' for an outlier or '1' for an inlier, even though the training data consists only of one class).

Then, in the documentation of average_precision_score you can see that the input y_score can be a non-thresholded measure, like from decision_function. You could also tinker with this, perhaps taking log of that score, etc., if you have any domain knowledge about it that gives you a reason to think to try it.

Once you have these manually created scores, you can pass them in for any of the precision / recall functions that need to vary the threshold. It's not perfect, but at least gives you a sense of how well the decision boundary is used for the classification.