CCAR-F - Prompt Engineering & Structured Output (20% of the exam) - Section 4.6
Design multi-instance and multi-pass review architectures.
A model retains its own reasoning context from generation, so it is less likely to question its own decisions in the same session. An independent review instance without that context catches subtle issues that self-review instructions and extended thinking do not. Candidates should split a large multi-file review into focused per-file passes for local issues plus a separate integration pass for cross-file data flow, and run verification passes that self-report confidence to enable calibrated routing.
self-review limitationsindependent review instanceper-file versus integration passesattention dilution and contradictory findingsconfidence-calibrated review routing
Harbourline Payments routes the output of their CI review instance by asking it to label every finding high or low confidence, posting the high-confidence findings as blocking comments and discarding the rest to protect the merge queue. A quarterly audit finds that 31 per cent of the discarded low-confidence findings described real defects, while blocking comments that developers dismissed as noise rose to 24 per cent. The architect is asked what those labels actually guarantee. What is the correct answer?
AThey are computed from the token probabilities behind the finding, so they measure how firmly the model settled on that wording and can be thresholded once the cut-off is tuned against the audit sample.
BThey are comparable between findings raised in different files and different passes, so a fixed threshold ranks defects consistently across a pull request once the prompt wording is held stable.
CThey are dependable within a single repository once the reviewer has seen enough of its code, because the labels calibrate themselves against the defect history the pass reads out of the version control log.
DThey report how the finding was expressed rather than a calibrated probability that the defect is real, so routing on them inherits that gap and discards real defects while promoting weak ones.check_circle Correct
Self-reported confidence is generated text, not a calibrated probability, so routing review findings on it inherits an uncontrolled error rate. A confidence label emitted with a finding is conditioned on the same context as the finding itself, so it expresses how the claim was phrased rather than how likely it is to hold. Routing decisions built on it therefore carry an error rate nobody has measured. Calibration has to come from something outside the pass, such as agreement between independent instances or a check against the code that can be run.
Why A is wrong: Tempting because sampling probabilities do exist and sound like the natural source of a confidence label. It is wrong because a label requested in the response is generated as text alongside the finding, not read out of the sampler, so tuning a cut-off tunes a stated word rather than a measurement.
Why B is wrong: Tempting because a shared prompt does make outputs look commensurable and consistency is what a routing threshold needs. It is wrong because each label is produced against its own local context, so the same word carries different meaning between passes and gives the threshold nothing stable to sort on.
Why C is wrong: Tempting because grounding a judgement in a repository's own history is a genuinely good review technique. It is wrong because a pass does not accumulate calibration between runs, and reading past commits changes what evidence a finding rests on without turning a stated label into a measured rate.
Why D is correct: Correct. A self-reported label is another piece of generated text conditioned on the same context that produced the finding, so it tracks the phrasing rather than the underlying likelihood, which is exactly the split the audit measured.
Exam traps in Prompt Engineering & Structured Output
Answers that look right on this material and are not. Each one is a distractor from a different question in the CCAR-F bank for this domain.
Extend the self-review prompt with a checklist of the eight defect classes the human reviewers rejected on, and require the authoring session to answer every checklist item before it posts.
errorWhy it is wrong: A defect taxonomy drawn from real rejections looks like a targeted fix, but the checklist is answered by the session that already concluded the patch works, so its stated justification is graded rather than the code, and the same blind spot survives the extra questions.
Move the job onto a configuration with a context window large enough to hold the entire diff comfortably, so that no part of the change falls outside what one review pass can read.
errorWhy it is wrong: The reasoning confuses capacity with attention, and the stem already shows the whole diff being read rather than truncated. Fitting more material into one context does not stop later material competing with earlier material, so findings stay concentrated at the front.
Keep both instances posting as they do now, and add a line to the pull request template asking the author to decide which of two conflicting automated comments applies to the hunk.
errorWhy it is wrong: Pushing arbitration to the author is honest about the disagreement, but it moves work onto the person the job exists to help and leaves the reviewer carrying the same unresolved output, so the acted-on rate has no reason to recover.