Typically, overfitting results in some weights being very high relative to others. To understand that, let's look at the histogram of weights that are obtained by running the model on the artificially created dataset in scenario 1:
We see that there are some weights that have a high value (> 0.1) and a majority that are centered around zero.
Let's now explore the impact of penalizing for having a high weight value through L1 and L2 regularizations.
The intuition of regularization is as follows:
- If the weight values are shrunk to as minimal as possible, it is less likely that some of those weights contribute more towards ...
