CCAR-F - Context Management & Reliability (15% of the exam) - Section 5.5
Design human review workflows and confidence calibration.
An aggregate accuracy figure can mask poor performance on one document type or one field, so accuracy must be validated by segment before high-confidence extractions are automated. Candidates should run stratified random sampling of high-confidence output to measure ongoing error rates and catch novel patterns, have the model emit field-level confidence scores, calibrate the review threshold against a labelled validation set, and route ambiguous or contradictory sources to limited reviewer capacity first.
A team is calibrating the review routing on its Claude Code workflow. They build a labelled set from the 300 changes that reviewers escalated over the past year, label each one as sound or defective, and fit a confidence threshold on that set. In a pilot against the escalated backlog the threshold looked accurate, but once applied to live traffic it sends 40 percent of all generated changes to review, several times what the pilot predicted, and the escaped defect rate is unchanged after six weeks. What should the architect do?
ARebuild the labelled set from a stratified random sample of all generated changes, escalated or not, label those independently of the original review decision, and refit the threshold on that sample.check_circle Correct
BEnlarge the labelled set by going back a further two years of escalated changes, so the threshold is fitted on roughly a thousand examples rather than three hundred and the resulting estimate becomes more stable.
CLower the fitted threshold until the share of live traffic entering review falls to the level the pilot predicted, then hold it there and monitor the escaped defect rate over a further quarter before revisiting it.
DWeight the escalated examples by how often each category of change appears in live traffic, and refit the threshold on that reweighted set without collecting any further labels.
Calibrate a routing threshold on a sample drawn from the population it will run against, labelled independently of the original routing decision. A set assembled from escalated cases is conditioned on the outcome being predicted, so its defect rate and its score distribution both differ from live traffic, and a threshold fitted on it is mis-sited when applied to the full population. Drawing a stratified random sample of all changes restores the base rate, and labelling those changes independently of the prior escalation decision stops the routing simply relearning existing reviewer behaviour.
Why A is correct: Correct: fitting a threshold requires a set drawn from the population the threshold will run against, and independent labelling breaks the circularity of learning from the very reviewer decisions the routing is meant to replace.
Why B is wrong: More examples narrow the confidence interval around an estimate, but they do not move it, so a larger sample drawn from the same escalated population reproduces the same skewed defect rate with greater apparent precision.
Why C is wrong: Tuning the threshold to hit a volume target treats the symptom rather than the fitting error, and because the underlying score was calibrated on an unrepresentative population, the changes it sheds are not the ones selected by risk.
Why D is wrong: Reweighting is the right instinct and does fix category proportions, but the escalated set holds no examples of changes that were never escalated, so no weighting can recover the behaviour of the score on the population it will mostly meet.
Answers that look right on this material and are not. Each one is a distractor from a different question in the CCAR-F bank for this domain.
It tells them the fax supplier's documents were extracted at roughly 98.4 percent accuracy as well, because a field-level metric is computed per field rather than per document and therefore already accounts for differences between sources.
errorWhy it is wrong: Tempting because field-level scoring is genuinely finer grained than document-level scoring, so it feels segment-aware. It is not: the field scores are still pooled into one mean across all sources, and pooling over a field dimension says nothing about variation over a source dimension.
The routing threshold sits too low for this document mix, so raising it to 0.95 would pull the faint carbon-copy quantities into review and let the existing confidence mechanism work as it was designed to.
errorWhy it is wrong: Tempting because moving a threshold is the obvious lever and would indeed capture more of the faint class. It cannot work here: the faint class scores higher than the accurate class, so any threshold that catches it catches the clean class first and floods the queue.
The validation set has grown too small relative to current monthly volume, so its estimates now carry too much sampling variance to hold a stable catch rate, and enlarging it to match the throughput would restore the original behaviour.
errorWhy it is wrong: Sample size genuinely bounds the precision of an estimate, so this reads as a statistical answer. Variance would make the catch rate wobble around 94 percent rather than move to a sustained lower level, and a larger set drawn from typed forms alone still omits the handwritten class.