In a feature extraction approach, we train only the top level of the network; the rest of the network remains fixed. Consider a feature extraction approach when the new dataset is relatively small and similar to the original dataset. In such cases, the higher-level features learned from the original dataset should transfer well to the new dataset.
Consider a fine-tuning approach when the new dataset is large and similar to the original dataset. Altering the original weights should be safe because the network is unlikely to overfit the new, large dataset.
Let us consider a pre-trained convolutional neural network, as shown in the following diagram. Using this we can study how the transfer of knowledge can be used ...
