Chapter 03: Supervised Classification

This chapter treats the supervised classification task in more detail. We will see examples of binary and multiclass classification and the difference of the discriminative and the generative approach. Especially, we will treat logistic regression, linear and quadratic discriminant analysis, naive bayes and k-NN classification.

Chapter 3.1: Classification Tasks

In classification, the task is to predict a categorical (binary or multiclass) label. In this section, we illustrate the concept of classification with some typical examples.

Chapter 3.2: Basic Definitions

Unless a classifiers should predict a discrete output in the end, classification models usually output scores or probablities first. We will explain why, introduce the concepts of decision regions and decision boundaries and finally differentiate two fundamental approaches to construct classifiers: the generative approach and the discriminant approach.

Chapter 3.3: Linear Classifiers

Linear classifiers are an essential subclass of classification models. This section provides the definition of a linear classifier and depicts differences between linear and non-linear decision boundaries.

Chapter 3.4: Logistic Regression

Logistic regression is a discrimant approach for constructing a classifier. We will motivate logistic regression via the logistic function, define the log loss for optimization and illustrate the approach in 1D and 2D.

Chapter 3.5: Discriminant Analysis

Discriminant analysis is a generative approach for constructing a classifier. We distinguish between linear (LDA) and quadratic (QDA) discriminant analysis where the latter is a more flexible approach.

Chapter 3.6: Naive Bayes

Naive Bayes is a generative approach for constructing a classifier and closely related to LDA and QDA.

Chapter 3.7: K-Nearest Neighbors

This section introduces k-nearest neighbors classification. We will explain in which sense this approach is fundamentally different to the previous sections and illustrate the effect of the hyperparameter k.