Regularization is another way to control overfitting, that penalizes individual weights in the model as they grow larger. If you're familiar with linear models such as linear and logistic regression, it's exactly the same technique applied at the neuron level. Two flavors of regularization, called L1 and L2, can be used to regularize neural networks. However, because it is more computationally efficient L2 regularization is almost always used in neural networks.
Quickly, we need to first regularize our cost function. If we imagine C0, categorical cross-entropy, as the original cost function, then the regularized cost function would be as follows:
Here, ; is a regularization parameter that can be increased ...
