Harbourline Payments routes the output of their CI review instance by asking it to label every finding high or low confidence, posting the high-confidence findings as blocking comments and discarding the rest to protect the merge queue. A quarterly audit finds that 31 per cent of the discarded low-confidence findings described real defects, while blocking comments that developers dismissed as noise rose to 24 per cent. The architect is asked what those labels actually guarantee. What is the correct answer?
- AThey are computed from the token probabilities behind the finding, so they measure how firmly the model settled on that wording and can be thresholded once the cut-off is tuned against the audit sample.
- BThey are comparable between findings raised in different files and different passes, so a fixed threshold ranks defects consistently across a pull request once the prompt wording is held stable.
- CThey are dependable within a single repository once the reviewer has seen enough of its code, because the labels calibrate themselves against the defect history the pass reads out of the version control log.
- DThey report how the finding was expressed rather than a calibrated probability that the defect is real, so routing on them inherits that gap and discards real defects while promoting weak ones. Correct
Why A is wrong: Tempting because sampling probabilities do exist and sound like the natural source of a confidence label. It is wrong because a label requested in the response is generated as text alongside the finding, not read out of the sampler, so tuning a cut-off tunes a stated word rather than a measurement.
Why B is wrong: Tempting because a shared prompt does make outputs look commensurable and consistency is what a routing threshold needs. It is wrong because each label is produced against its own local context, so the same word carries different meaning between passes and gives the threshold nothing stable to sort on.
Why C is wrong: Tempting because grounding a judgement in a repository's own history is a genuinely good review technique. It is wrong because a pass does not accumulate calibration between runs, and reading past commits changes what evidence a finding rests on without turning a stated label into a measured rate.
Why D is correct: Correct. A self-reported label is another piece of generated text conditioned on the same context that produced the finding, so it tracks the phrasing rather than the underlying likelihood, which is exactly the split the audit measured.