NCA-ADS - Machine Learning With RAPIDS (16% of the exam) - Section 2.3
Evaluate models.
Evaluate trained models using held-out test sets and appropriate scoring functions to estimate generalisation performance. Recognise the risk of data leakage when the test set influences preprocessing or threshold selection.
Practice question for this objective
lock_openFree sampleMachine Learning With RAPIDSmedium
An engineer uses cross-validation to choose between several hyperparameter settings for a GPU-accelerated XGBoost model, then reports the best mean cross-validation score as the model's expected performance on new data. A reviewer notes that this reported figure is likely to be optimistic. Which experimental design corrects this optimism while still using cross-validation to tune the hyperparameters?
AHold out a separate test set before tuning, run cross-validation only on the remaining data to choose hyperparameters, then evaluate the chosen model once on the untouched test set, or equivalently use nested cross-validation.check_circle Correct
BRepeat the same five-fold cross-validation several times with different random seeds and average the best scores, so that the optimism is removed by averaging out the random fold assignment.
CUse leave-one-out cross-validation instead of five-fold for the tuning step, because training on nearly all of the data each time eliminates the optimistic bias in the selected score.
DReport the mean of all configurations' cross-validation scores rather than the best one, because the average across configurations is an unbiased estimate of the selected model's performance.
The best cross-validation score across many hyperparameter settings is optimistically biased; estimate generalisation on data not used for selection, via a held-out test set or nested cross-validation. Selecting the configuration with the highest cross-validation score means that score has been maximised over the noise of the folds, so it is biased upward as an estimate of true performance. An outer evaluation that never participates in the selection, either a final untouched test set or the outer loop of nested cross-validation, isolates tuning from assessment and yields an unbiased generalisation estimate.
Why A is correct: Because the best cross-validation score is the maximum over many configurations, it is biased upward by selection; evaluating the chosen model on data that played no part in selection, via a held-out test set or an outer nested loop, gives an unbiased estimate of generalisation.
Why B is wrong: Repeated cross-validation reduces the variance of the estimate, which makes it look like a fix, but it still reports the score of the configuration that was selected on those same folds, so the selection optimism remains regardless of how many repeats are averaged.
Why C is wrong: Leave-one-out lowers bias in estimating a single model's error but does nothing about the optimism that comes from picking the best of many configurations on the same data, and it is computationally heavy, so it neither targets nor fixes the described problem.
Why D is wrong: Averaging over configurations describes a typical configuration, not the one actually chosen, so it answers a different question and tends to understate the selected model rather than give its honest generalisation estimate, making it a plausible but invalid substitute.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Gradient-boosted models cannot be tuned against a single test set at all, because boosting requires a dedicated validation set for early stopping, so the procedure cannot run.
errorWhy it is wrong: Tempting because boosting does benefit from a validation set for early stopping, but the procedure can run and produce numbers; the genuine defect is the biased selection, not an inability to execute it.
cuML's StandardScaler does not support fitting on datasets larger than GPU memory, so the combined fit will raise an out-of-memory error in most practical scenarios.
errorWhy it is wrong: cuML's StandardScaler can handle large datasets through Dask-cuML or chunked processing; memory limits are a practical concern but not the fundamental reason this workflow is flawed. The core issue is statistical, not a memory constraint.
Because the validation set is always smaller than the training set, its accuracy estimates carry more sampling variance and are therefore systematically too high.
errorWhy it is wrong: A small validation set raises variance in both directions, not a systematic upward bias; the real issue is selection over many candidates, so this confuses variance with the selection bias being described.