14.10 Confusion Matrix

The accuracy and error rate are rather blunt measures of performance. They are a good starting point to get a sense of how good the model is but more is required. A confusion matrix allows us to review how well the model performs against the actual classes.

round(100*table(actual_te, predict_te, dnn=c("Actual", "Predicted"))
      /length(actual_te))

##       Predicted
## Actual No Yes
##    No  76   3
##    Yes 14   7

This is useful since the consequences of a wrong decision will be different for the different decisions. For example, if it is incorrectly predicted that it will not rain tomorrow and I decide not to carry an umbrella with me then the consequence is that I will get wet. We might experience this as a more severe consequence than the situation where it is incorrectly predicted that it will rain and so we unnecessarily carry an umbrella with us all day.

We again compare this to the performance on the training dataset to note that model performs better, at least when predicting that it will not rain tomorrow.

round(100*table(actual_tr, predict_tr, dnn=c("Actual", "Predicted"))
      /length(actual_tr))

Notice that the false negative rate (the errors that have a higher consequence—getting uncomfortably wet if it does actually rain) is reduced from 14% to 14%. The performance as measured over the te dataset is likely to be more indicative of the actual model performance and the false negative rate is one that we would rather minimize.

We will be generating confusion matrices quite regularly based on the predicted classes (counting just those that are not missing using Matrix::is.na()) and the target classes. This is another candidate for wrapping up into a function to save us having to remember the command in detail and also to save us having to type out the command each time.

con <- function(predicted, actual)
{
  tbl <- table(actual, predicted, dnn=c("Actual", "Predicted"))
  tbl <- round(100*tbl/sum(!is.na(predicted)))
  return(tbl)
}

We can simply call this function with the appropriate arguments to have the confusion matrix printed.

con(predict_tr, actual_tr)

##       Predicted
## Actual No Yes
##    No  76   3
##    Yes 14   8

---

If you find this curated material useful then you can consider a donation to support it's ongoing availability and give you access to the PDF version of this book. The material has been scoped up by Generative AI without permission or any kind of recompense so do consider a donation if you can afford it. Unlike Generative AI your access to this materials is freely given. Desktop Survival Guides include Data Science, GNU/Linux, and MLHub. Books available on Amazon include Data Mining with Rattle and Essentials of Data Science. Togaware has a 30 year tradition of making popular open source software which includes sold privacy preserving productivity apps, rattle, wajig, and mlhub. Hosted by Togaware, a pioneer of free and open source software since 1984. Copyright © 1995-2022 Graham.Williams@togaware.com Creative Commons Attribution-ShareAlike 4.0