A team is calibrating the review routing on its Claude Code workflow. They build a labelled set from the 300 changes that reviewers escalated over the past year, label each one as sound or defective, and fit a confidence threshold on that set. In a pilot against the escalated backlog the threshold looked accurate, but once applied to live traffic it sends 40 percent of all generated changes to review, several times what the pilot predicted, and the escaped defect rate is unchanged after six weeks. What should the architect do?
- ARebuild the labelled set from a stratified random sample of all generated changes, escalated or not, label those independently of the original review decision, and refit the threshold on that sample. Correct
- BEnlarge the labelled set by going back a further two years of escalated changes, so the threshold is fitted on roughly a thousand examples rather than three hundred and the resulting estimate becomes more stable.
- CLower the fitted threshold until the share of live traffic entering review falls to the level the pilot predicted, then hold it there and monitor the escaped defect rate over a further quarter before revisiting it.
- DWeight the escalated examples by how often each category of change appears in live traffic, and refit the threshold on that reweighted set without collecting any further labels.
Why A is correct: Correct: fitting a threshold requires a set drawn from the population the threshold will run against, and independent labelling breaks the circularity of learning from the very reviewer decisions the routing is meant to replace.
Why B is wrong: More examples narrow the confidence interval around an estimate, but they do not move it, so a larger sample drawn from the same escalated population reproduces the same skewed defect rate with greater apparent precision.
Why C is wrong: Tuning the threshold to hit a volume target treats the symptom rather than the fitting error, and because the underlying score was calibrated on an unrepresentative population, the changes it sheds are not the ones selected by risk.
Why D is wrong: Reweighting is the right instinct and does fix category proportions, but the escalated set holds no examples of changes that were never escalated, so no weighting can recover the behaviour of the score on the population it will mostly meet.