During hyperparameter tuning of a cuML gradient-boosted model, a team evaluates each candidate configuration by measuring its accuracy on the held-out test set and selecting the configuration with the highest test accuracy. What fundamental problem does this practice introduce?
- AIt makes training slower because the test set is larger than a validation set would be
- BIt causes the chosen hyperparameters to overfit to the test set, so the reported accuracy is an optimistic and unreliable estimate of true generalisation Correct
- CIt prevents the model from learning the correct feature weights because test labels should never be loaded into memory during training
- DIt forces the search algorithm to treat all hyperparameter combinations as equally likely, removing any benefit from Bayesian strategies
Why A is wrong: The size of the evaluation set does not drive the fundamental methodological problem here; the concern is about information leakage, not computational cost.
Why B is correct: When the test set is used repeatedly to select hyperparameters, it effectively becomes part of the training process. The model configuration is chosen to maximise performance on that specific sample, inflating the reported metric and giving no unbiased estimate of how the model will behave on genuinely unseen data.
Why C is wrong: Test labels are not fed into the model during forward or backward passes when used purely for evaluation; the issue is statistical leakage through repeated selection decisions, not label exposure during gradient computation.
Why D is wrong: The choice of which dataset split is used for evaluation is independent of whether the search strategy is Bayesian, random, or grid-based; using the test set for selection is a data-protocol error, not a search-algorithm constraint.