TensorFlow Machine Learning Cookbook - Second Edition

There's more...

To understand what is happening, we should look at how a large learning rate and small learning rate act on L1 norms and L2 norms. To visualize this, we look at a one-dimensional representation of learning steps on both norms, as follows:

Figure 7: What can happen with the L1 and L2 norm with larger and smaller learning rates