CCAR-P domain - 16% of the exam

Evaluation, Testing & Optimization

Evaluation, Testing & Optimization is 16% of the CCAR-P exam. These are the objectives it covers, each with practice questions, with every answer explained.

The domain in numbers

51
Practice questions
6
Objectives
34
Medium
17
Hard

Where people go wrong

  • Measuring an average instead of the named failure

    The wrong answers reach for a single overall figure: a similarity score, overall accuracy, a mean score or one blended score. These averages are dominated by the routine cases, so a dropped allergy line, a missed credential or a fall in emergency escalations barely moves them, and a higher threshold or larger sample leaves the problem in place. The correct answers measure the named failure directly: recall on labelled records, a separate must-pass gate on the slice, or a slice reported apart.

    See it in a sample question
  • Using batch processing to fix latency

    The wrong answers offer batch processing as a cost or latency fix, but batch requests complete asynchronously with no promise of a fast turnaround. Where someone is waiting, such as a resident expecting a text reply, a caller on the line or a pharmacist at the counter, or where a review comment must post within four seconds at p95, batching makes the wait longer. Latency falls by shrinking the output, trimming the input to what answers use, or caching a repeated prefix.

    See it in a sample question
  • Repairing a stale evaluation set by making it bigger

    The wrong answers try to fix an evaluation set that no longer matches live traffic by making it bigger, resampling the same period or inventing cases. A larger set drawn from the same old traffic still holds no disputes from the new product or questions from the new courses, and synthetic cases reflect the generator's assumptions. A set the prompt authors can read also rewards fit to known cases. The correct answers use held-out, freshly sampled or shadowed live traffic.

    See it in a sample question

Objectives in this domain

What to study

The official documents this domain's practice questions cite most.

Sample questions from this domain

Free sampleEvaluation, Testing & Optimizationmedium

A hospital group is piloting Claude to draft discharge summaries from inpatient records, and the clinical safety officer has set one requirement in writing: no summary may leave out a documented drug allergy or a medication change. The team's evaluation plan scores each draft against a clinician-written reference summary using an overall similarity score, and the pilot average is 0.87. About one record in six carries an allergy or a medication change. What should the architect recommend the team measure before the pilot widens?

  • AThe overall similarity score, with its pass threshold raised from 0.87 to 0.92 across the pilot
  • BThe share of summaries the model itself rates as complete, from a self-check added to each draft
  • CRecall of documented allergies and medication changes per summary, on a labelled set, reported apart Correct
  • DClinician satisfaction with each draft, as a five-point rating from the ward doctors who sign it
Derive the metric from the stated requirement, measuring recall on safety-critical items separately so an aggregate quality score cannot hide omissions. The requirement is defined by a failure on specific items, so the metric has to count those items. An overall similarity score averages over all text and over records that contain no allergy or medication change, which lets a 0.87 mean coexist with missed allergies. Item-level recall on a labelled set, with a threshold tied to the safety officer's requirement and reported separately, exposes exactly the failure the requirement forbids.

Why A is wrong: Raising the bar is tempting because it looks like a stricter safety gate. It is wrong because the similarity score averages over the whole narrative and over the five in six records with no allergy or medication change, so a dropped allergy line barely moves it and a higher threshold still does not target the stated requirement.

Why B is wrong: A self-check is tempting because it is automatic and cheap to run on every draft. It is wrong because a model's own judgement of completeness is not a ground-truth accuracy signal; the same model that omitted an allergy can rate its draft complete.

Why C is correct: The requirement names a specific failure, the omission of an allergy or medication change, so the metric must count those items: recall against records labelled with them, reported separately from any aggregate so a good average cannot hide a miss.

Why D is wrong: Clinician ratings are tempting because the raters are the domain experts. It is wrong because a satisfaction score reflects overall impression and readability; a busy signer does not check every line against the record, so omissions go unmeasured.

Free sampleEvaluation, Testing & Optimizationmedium

A hospital pharmacy runs an assistant that answers pharmacists' dose-check questions while they verify a prescription at the counter, with a stated budget of five seconds per answer that the service currently misses. Each request sends the patient's entire electronic record, averaging about 90,000 tokens. A review of 250 answered questions, which the team keeps as a labelled eval set, found that every answer drew on the current medication list, recorded allergies, renal function and the latest laboratory results, and nothing else. Most patients are the subject of a single query. Which change best fits the stated constraints?

  • AKeep sending the full record and enable prompt caching so repeated content is processed faster
  • BTruncate each record to its most recent 10,000 tokens so that older entries drop out of every request
  • CBuild each request from the medication list, allergies, renal function and latest results, then re-run the eval set Correct
  • DSubmit the dose-check questions through the Message Batches API so the large records cost less to process
Trim context to the sections a task demonstrably uses, selected by relevance rather than position, and confirm quality with an eval re-run. Input size drives both processing time and cost, so removing tokens the answers never use is the direct fix for a latency budget broken by a 90,000-token record. Selecting by relevance keeps every section the review proved necessary, including old allergy entries, and the labelled eval set gives evidence that the trimmed context still produces correct dose checks.

Why A is wrong: Tempting because caching is the standard answer to large prompts. It is wrong because each record belongs to a different patient and most are queried once, so there is rarely a repeated prefix to reuse and the full record is still processed every time.

Why B is wrong: Tempting because recency is a simple, cheap rule for trimming context. It is wrong because it cuts by position rather than relevance, so allergies recorded years earlier can be lost, which removes context a dose check requires.

Why C is correct: Correct. The review shows these four sections are everything the answers use, so sending only them cuts input tokens and processing time sharply, and re-running the labelled set confirms answer quality holds after the trim.

Why D is wrong: Tempting because batch processing reduces the cost of large requests. It is wrong because the pharmacist is waiting at the counter against a five-second budget, and asynchronous batch processing makes answers slower, not faster.

Free sampleEvaluation, Testing & Optimizationhard

A city council's permit enquiry assistant has been through 20 prompt revisions, each judged against the same 300 labelled enquiries, which every prompt author can read. The eval score has risen from 71 to 94 percent, but the share of live enquiries escalated to staff has not moved. The team can afford to label 200 new enquiries this quarter and needs an eval score it can trust for release decisions. What should the architect recommend?

  • AAdd the 200 new enquiries to the existing 300 so that authors tune against a larger set, which makes the score harder to overfit by chance
  • BSwitch the 300 enquiries from code-graded to model-graded scoring, since a model grader judges intent and is harder to tune prompts against
  • CLabel the 200 new enquiries as a held-out set that authors never see, use it only for release decisions, and keep the 300 for development Correct
  • DShip the next revision only after a live A/B test, choosing the winner from the first fifty escalation decisions in each arm of the test
Repeatedly tuning against a visible eval set overfits it, so release decisions need a held-out set the prompt authors never see. Each revision was chosen because it scored better on the same 300 items, so the prompt gradually encoded those items' quirks and the score drifted away from live behaviour. Splitting off a held-out set that is used only for release decisions breaks that feedback loop: authors iterate on the development set, and the held-out score estimates how a revision performs on enquiries it was not tuned to.

Why A is wrong: This is tempting because a larger set does reduce noise. It is wrong because authors would still read and tune against every item, so the combined set becomes another development set and the release score keeps rewarding fit to known cases.

Why B is wrong: This is tempting because model grading can capture intent that rigid checks miss. It is wrong because the problem is that the test items are known to the authors, not how they are scored; changing the grader leaves the overfitting in place.

Why C is correct: This is correct because a rising score on items the authors can read measures fit to those items, not general quality. A held-out set kept away from authors gives an unbiased estimate of live performance and fits the labelling budget.

Why D is wrong: This is tempting because live traffic is the real target. It is wrong because fifty decisions per arm is far too small a sample to separate a real difference from noise, and it exposes residents to untested revisions instead of fixing the offline measure.

Other domains in this exam

See also the CCAR-P cert hub, the study guide, and the cheat sheet.

Examworthy is not affiliated with or endorsed by Anthropic. Original, blueprint-aligned practice material only.