CCDV-F - Eval, Testing, and Debugging (2.6% of the exam) - Section 4.1
Debug Claude applications and handle errors with an appropriate recovery strategy.
Published skill weight 2.6 percent, and this skill is the whole domain. Covers error type identification, recovery strategy selection, trace analysis to identify failure modes, and isolating whether a problem originates in the integration layer or in the model output itself, which is the discrimination this skill turns on.
error type identificationrecovery strategy selectiontrace analysisintegration layer versus model output
Practice question for this objective
lock_openFree sampleEval, Testing, and Debuggingmedium
A logistics assistant answers delivery questions by calling one internal tracking tool. Support has measured that 6 per cent of conversations end with a confident delivery date that the carrier system never supplied. A stored trace for one such conversation is reproduced below. The team must attribute the failure to a specific layer before proposing a fix. What does the trace establish?
turn 2 assistant tool_use track_shipment {"ref": "SH-40182"}
turn 3 user tool_result is_error=true
content: "upstream timeout after 30000 ms"
turn 4 assistant text "Your parcel is due on Thursday."
turn 4 stop_reason: end_turn
AThe model ignored the tool it was given, so the tool description should be rewritten to state more firmly that the tracking tool must be called before any delivery date is given.
BThe generation was cut short by the output limit, so the integration should raise the token ceiling on the request and replay the conversation to obtain the full answer.
CThe tracking dependency timed out and the integration passed that failure back as an ordinary tool result, so the recovery path for a failed tool call is the layer at fault rather than the model's reasoning.check_circle Correct
DThe model invented a delivery date without provocation, so the sampling temperature on the request should be lowered until the fabricated dates stop appearing in production traffic.
Read a trace before blaming the model: an errored tool result handed back as ordinary content is an integration-layer fault. The trace fixes the sequence: the tool was invoked, the dependency timed out, and the failure was returned to the model as a routine tool result with no application-side handling. Once a failed call is presented as just another result, the model has nothing authoritative to answer from and falls back on conversation text. The defect therefore sits in the error-recovery path of the integration layer, which should surface an explicit unavailable state rather than let an unverified answer proceed.
Why A is wrong: Tempting because tool descriptions do influence tool selection, but the trace shows the tool WAS called at turn 2, so a description rewrite treats a fault the trace has already ruled out.
Why B is wrong: Tempting because truncation is a common cause of odd endings, but the recorded stop reason is a normal end of turn rather than a limit, so no truncation occurred.
Why C is correct: Correct: the trace records an errored tool result carrying a timeout, and the application had no rule for what to do when a tool fails, so the model was left to answer from conversation text.
Why D is wrong: Tempting because the answer is indeed unsupported, but sampling settings do not decide what happens when a dependency fails, and the invented date has a recorded upstream cause.
Answers that look right on this material and are not. Each one is a distractor from a different question in the CCDV-F bank for this domain.
Keep the fixed five attempts for every failure class, but shorten the gap between attempts to 200 milliseconds so all five fit inside the four second budget.
errorWhy it is wrong: Tempting because it does bring the attempts inside the budget, but it retries the permanently failing item code just as hard, and a shorter gap during an overload window adds load precisely when the dependency needs relief.
The rejections indicate degraded answers under load, so the request should carry a smaller output ceiling to shorten each generation and free capacity for the remaining documents in the batch.
errorWhy it is wrong: Tempting because a lower output ceiling does shorten generations, but a throughput rejection happens before any generation and the change would truncate answers while leaving the retry storm untouched.
Tighten the input schema further by expressing the maximum adjustment as an upper bound on the amount field, so both remaining failure classes are prevented before the arguments are produced.
errorWhy it is wrong: Tempting because schema constraints genuinely reduce malformed arguments, but the limit depends on the individual employee's salary rather than one fixed number, and a schema still cannot force a tool call to be made at all.