At this point, Amazon ML will use the training set to train several models and the evaluation sets to select the best one.
Amazon ML runs several model training in parallel, each time trying new parameters and shuffling the training set at each new pass. Once the number of passes initially set has been exhausted or the algorithm has converged, whichever comes first, the model is considered trained. For each model it trains, Amazon ML uses it for prediction on the validation subset to obtain an evaluation score per model. Once all the models have been trained and evaluated this way, Amazon ML simply selects the one with the best evaluation score.
The evaluation metric depends on the type of prediction at hand. AUC and ...
