NCA-ADS - Machine Learning With RAPIDS (16% of the exam) - Section 2.5
Apply cross-validation methods.
Apply k-fold and stratified cross-validation to obtain robust performance estimates on limited datasets. Distinguish between cross-validation for model selection and the separate hold-out test set required for unbiased final evaluation.
Practice question for this objective
lock_openFree sampleMachine Learning With RAPIDSmedium
During hyperparameter tuning of a cuML gradient-boosted model, a team evaluates each candidate configuration by measuring its accuracy on the held-out test set and selecting the configuration with the highest test accuracy. What fundamental problem does this practice introduce?
AIt makes training slower because the test set is larger than a validation set would be
BIt causes the chosen hyperparameters to overfit to the test set, so the reported accuracy is an optimistic and unreliable estimate of true generalisationcheck_circle Correct
CIt prevents the model from learning the correct feature weights because test labels should never be loaded into memory during training
DIt forces the search algorithm to treat all hyperparameter combinations as equally likely, removing any benefit from Bayesian strategies
Explain why hyperparameter selection must be guided by a validation set rather than the test set to preserve an unbiased generalisation estimate. The test set is intended to provide a single, unbiased measurement of the final model's generalisation. Each time a decision - including hyperparameter selection - is made based on test-set feedback, information from that set leaks into the model-building process. Over many tuning iterations, the selected configuration is increasingly tailored to the quirks of the test sample, making the recorded accuracy optimistic. The correct approach is to tune against a separate validation set (or use cross-validation) and reserve the test set for one final evaluation after all configuration decisions have been made.
Why A is wrong: The size of the evaluation set does not drive the fundamental methodological problem here; the concern is about information leakage, not computational cost.
Why B is correct: When the test set is used repeatedly to select hyperparameters, it effectively becomes part of the training process. The model configuration is chosen to maximise performance on that specific sample, inflating the reported metric and giving no unbiased estimate of how the model will behave on genuinely unseen data.
Why C is wrong: Test labels are not fed into the model during forward or backward passes when used purely for evaluation; the issue is statistical leakage through repeated selection decisions, not label exposure during gradient computation.
Why D is wrong: The choice of which dataset split is used for evaluation is independent of whether the search strategy is Bayesian, random, or grid-based; using the test set for selection is a data-protocol error, not a search-algorithm constraint.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Report only the highest fold accuracy of 0.90, because cross-validation is intended to reveal the best achievable performance of the model across the folds.
errorWhy it is wrong: Picking the maximum fold is tempting because it is the most flattering number, but cross-validation is not a search for the luckiest fold; reporting the best fold overstates expected performance and ignores the variability the procedure is meant to expose.
Increase the number of folds from five to twenty so that each fold is smaller and the random variation between folds averages out across more repetitions.
errorWhy it is wrong: More folds change the bias-variance trade-off of the estimate but make each fold smaller, which with a 3 per cent positive rate makes empty or near-empty positive folds more likely rather than less, so this worsens the instability described.
Take the single best fold score for each configuration, so each is judged on its strongest split.
errorWhy it is wrong: Choosing the best fold rewards lucky splits and maximises optimism; it appears to surface the strongest configuration but systematically overestimates generalisation.