Data Mining Algorithms: Explained Using R

Chapter 7

Classification model evaluation

7.1 Introduction

The purpose of the evaluation of a classification model is to get a reliable assessment of the quality of the target concept's approximation represented by the model, which will be called the model's predictive performance. Different performance measures can be used, depending on the intended application of the model. Given the fact that the model is created based on a training set, which is a usually small subset of the domain, it is its generalization properties that are essential for the approximation quality. For any performance measure, it is important to distinguish between its value for a particular dataset (dataset performance), especially the training set (training performance), and its expected performance on the whole domain (true performance).

7.1.1 Dataset performance

The dataset performance of a model is assessed by calculating the value of one or more selected performance measures on a particular dataset with true class labels available. It describes the degree of match between the model and the target concept on this dataset.

7.1.2 Training performance

Evaluating a model on the training set that was used to create the model determines the model's training performance. Whereas it is sometimes useful to better understand the model and diagnose the operation of the employed classification algorithm, it is usually not of significant interest, since the purpose of classification models is not to classify ...