Chapter 15: Regularization

This chapter introduces the concept of regularization and discusses common regularization techniques in more depth.

Chapter 15.01: Introduction to Regularization

In this section, we revisit overfitting and introduce regularization as a mechanism against overfitting.

Chapter 15.02: Lasso and Ridge Regression

In this section, we introduce Ridge and Lasso regression.

Chapter 15.03: Lasso vs. Ridge Regression

In this section, we compare Ridge against Lasso regression.

Chapter 15.04: Elastic Net and Regularization for GLMs

In this section, we introduce the elastic net as a combination of Ridge and Lasso regression and discuss regularization for logistic regression.

Chapter 15.05: Regularization for Underdetermined Problems

In this section, we discuss how regularization can make ill-posed problems well-defined.

Chapter 15.06: L0 Regularization

In this section, we introduce LQ regularization and particularly discuss L0 regularization.

Chapter 15.07: Regularization in Non-Linear Models and Bayesian Priors

In this section, we motivate regularization from a Bayesian perspective.

Chapter 15.08: Geometric Analysis of L2-Regularization and Weight Decay

In this section, we provide a geometric understanding of L2 regularization and discuss its correspondence to weight decay.

Chapter 15.09: Geometric Analysis of L1-regularization

In this section, we provide a geometric understanding of L1 regularization.

Chapter 15.10: Early Stopping

In this section, we introduce early stopping and show how it can act as a regularizer.