To improve on this, we basically have the following options:
- Add more data: Maybe it is just not enough data for the learning algorithm and we should simply add more training data?
- Play with the model complexity: Maybe the model is not complex enough? Or maybe it is already too complex? In this case, we could decrease k so that it would take less nearest neighbors into account and thus be better in predicting non-smooth data. Or we could increase it to achieve the opposite.
- Modify the feature space: Maybe we do not have the right set of features? We could, for example, change the scale of our current features or design even more new features. Or should we rather remove some of our current features in case some features are ...