The RMSE is a relative quantity and not an absolute one. An RMSE of 100 may be good for a certain context, while a RMSE of 10 may be a sign of poor predictions in another context. It is, therefore, important to have a baseline that we can compare our model to. Each Amazon ML evaluation provides a baseline, which is calculated differently depending on the nature of the problem (regression, binary or multiclass classification). The baseline is the score we would obtain using the most simple and obvious model:
- The baseline for regression is given by a model that always predicts the mean of the target
- The baseline for binary classification is an AUC of 0.5, which is the score for a model that randomly assigns 0 or 1 ...
