An engineer uses cross-validation to choose between several hyperparameter settings for a GPU-accelerated XGBoost model, then reports the best mean cross-validation score as the model's expected performance on new data. A reviewer notes that this reported figure is likely to be optimistic. Which experimental design corrects this optimism while still using cross-validation to tune the hyperparameters?
- AHold out a separate test set before tuning, run cross-validation only on the remaining data to choose hyperparameters, then evaluate the chosen model once on the untouched test set, or equivalently use nested cross-validation. Correct
- BRepeat the same five-fold cross-validation several times with different random seeds and average the best scores, so that the optimism is removed by averaging out the random fold assignment.
- CUse leave-one-out cross-validation instead of five-fold for the tuning step, because training on nearly all of the data each time eliminates the optimistic bias in the selected score.
- DReport the mean of all configurations' cross-validation scores rather than the best one, because the average across configurations is an unbiased estimate of the selected model's performance.
Why A is correct: Because the best cross-validation score is the maximum over many configurations, it is biased upward by selection; evaluating the chosen model on data that played no part in selection, via a held-out test set or an outer nested loop, gives an unbiased estimate of generalisation.
Why B is wrong: Repeated cross-validation reduces the variance of the estimate, which makes it look like a fix, but it still reports the score of the configuration that was selected on those same folds, so the selection optimism remains regardless of how many repeats are averaged.
Why C is wrong: Leave-one-out lowers bias in estimating a single model's error but does nothing about the optimism that comes from picking the best of many configurations on the same data, and it is computationally heavy, so it neither targets nor fixes the described problem.
Why D is wrong: Averaging over configurations describes a typical configuration, not the one actually chosen, so it answers a different question and tends to understate the selected model rather than give its honest generalisation estimate, making it a plausible but invalid substitute.