CCAR-F - Prompt Engineering & Structured Output (20% of the exam) - Section 4.1
Design prompts with explicit criteria to improve precision and reduce false positives.
Explicit categorical criteria beat vague instruction. Telling a reviewer to be conservative or to report only high-confidence findings does not improve precision the way a specific rule does, such as flagging a comment only where the claimed behaviour contradicts the actual behaviour. Candidates should define which issue classes to report and which to skip, set severity criteria with concrete code examples, and temporarily disable a high false-positive category while its prompt is repaired.
explicit categorical criteriaconfidence-based filtering limitsfalse positive rate and developer trustseverity criteria with examplesreview scope definition
A payments team runs Claude Code inside its CI pipeline to review every pull request on the settlement service, and the review prompt instructs it to flag any security concerns it finds. Over 260 pull requests, reviewers dismissed 41 percent of the posted comments as not actionable, most of them speculative remarks about code that handles no external input. The team must bring the false positive rate down without losing the genuine findings. Which change most effectively addresses the measured rate?
ARaise the acceptance threshold so that findings the model reports at high self-assessed confidence are the ones posted to the pull request.
BMove the review job to the Message Batches API so that a slower and more thorough pass can run against every open pull request overnight.
CReplace the open instruction with a named list of qualifying finding categories, each carrying one qualifying and one disqualifying example, and post nothing when no category matches.check_circle Correct
DAdd a second review pass with a separately trained classifier that scores each drafted comment for usefulness before that comment reaches the pull request.
An open instruction to flag concerns produces false positives; named categories with qualifying and disqualifying examples give the model a testable boundary. Precision follows from how narrowly the prompt defines a qualifying finding. An instruction to flag any security concern gives no membership test, so the model resolves the ambiguity by reporting anything it can argue for. Naming the categories, showing one example that qualifies and one that does not, and stating that no comment is the correct output when nothing matches converts a judgement call into a decision the model can apply consistently.
Why A is wrong: Tempting because a confidence gate is quick to add and appears to filter weak output, but a self-reported score is not calibrated against the objective criteria for a real defect, so speculative remarks that the model is sure about survive the gate while quiet genuine findings are cut.
Why B is wrong: Tempting because batch processing lowers cost and allows a longer pass, but the Message Batches API is asynchronous with a processing window of up to 24 hours, which cannot serve a review that must appear on the pull request during the build, and thoroughness is not the problem being measured.
Why C is correct: Correct because the false positives come from an instruction that defines no boundary for what counts as a security concern, and explicit categories with a matched pair of examples give the model a testable membership decision plus a sanctioned empty outcome.
Why D is wrong: Tempting because a dedicated model sounds like a stronger filter than prompt text, but it adds training and maintenance cost and a second inference stage inside a build-time budget before the cheap prompt fix has been tried, and it inherits the same undefined notion of a security concern.
Exam traps in Prompt Engineering & Structured Output
Answers that look right on this material and are not. Each one is a distractor from a different question in the CCAR-F bank for this domain.
The threshold sits too high for the review prompt in use, and moving it to 0.6 would readmit the suppressed defects while keeping the naming opinions below the line.
errorWhy it is wrong: Tuning a threshold is the obvious response, but the naming opinions scored 0.95 and the defects 0.72, so no threshold on this scale separates them; lowering the line admits more of both.
Tighten the definition of the high label in the review prompt, reserving it for findings the pass would be willing to stake a red build on, and give three examples of each label.
errorWhy it is wrong: Sharpening the rubric is a reasonable first instinct and would improve label wording, but the measurement shows the label carries no relationship to correctness, so redefining a self-reported judgement leaves the gate resting on the same unreliable signal.
Cap the number of posted comments at ten and let the model choose which ten to keep, so that reviewers are given a shorter list on every pull request.
errorWhy it is wrong: Tempting because a cap guarantees a readable thread and is trivial to implement, but truncation without a severity rule leaves the selection to the same undirected judgement that produced the ordering, so a blocker can be dropped to make room for a naming preference.