R package caret confusionMatrix with missing categories

Question

I am using the function confusionMatrix in the R package caret to calculate some statistics for some data I have. I have been put

Answer 1

First note that confusionMatrix can be called as confusionMatrix(predicted, actual) in addition to being called with table objects. However, the function throws an error if predicted and actual (both regarded as factors) do not have the same number of levels.

This (and the fact that the caret package spit an error on me because they don't get the dependencies right in the first place) is why I'd suggest to create your own function:

# Create a confusion matrix from the given outcomes, whose rows correspond
# to the actual and the columns to the predicated classes.
createConfusionMatrix <- function(act, pred) {
  # You've mentioned that neither actual nor predicted may give a complete
  # picture of the available classes, hence:
  numClasses <- max(act, pred)
  # Sort predicted and actual as it simplifies what's next. You can make this
  # faster by storing `order(act)` in a temporary variable.
  pred <- pred[order(act)]
  act  <- act[order(act)]
  sapply(split(pred, act), tabulate, nbins=numClasses)
}

# Generate random data since you've not provided an actual example.
actual    <- sample(1:4, 1000, replace=TRUE)
predicted <- sample(c(1L,2L,4L), 1000, replace=TRUE)

print( createConfusionMatrix(actual, predicted) )

which will give you:

      1  2  3  4
[1,] 85 87 90 77
[2,] 78 78 79 95
[3,]  0  0  0  0
[4,] 89 77 82 83

Answer 2

You can use union to ensure similar levels:

library(caret)

# Sample Data
predicted <- c(1,2,1,2,1,2,1,2,3,4,3,4,6,5) # Levels 1,2,3,4,5,6
reference <- c(1,2,1,2,1,2,1,2,1,2,1,3,3,4) # Levels 1,2,3,4

u <- union(predicted, reference)
t <- table(factor(predicted, u), factor(reference, u))
confusionMatrix(t)

Answer 3

I had the same problem and here is my solution:

tab <- table(my_prediction, my_real_label)
if(nrow(tab)!=ncol(tab)){

missings <- setdiff(colnames(tab),rownames(tab))

missing_mat <- mat.or.vec(nr = length(missings), nc = ncol(tab))
tab  <- as.table(rbind(as.matrix(tab), missing_mat))
rownames(tab) <- colnames(tab)
}

my_conf <- confusionMatrix(tab)

Cheers Cankut