Support debugging and operational issue resolution - CCAR-P Developer Productivity & Operational Enablement

Support debugging and operational issue resolution.

Using Claude and system telemetry to debug production issues: reading traces and logs, isolating whether a fault lies in the integration or the model output, and resolving it without guessing.

trace and log analysisintegration versus model faultroot-cause isolation

Practice question for this objective

Free sampleDeveloper Productivity & Operational Enablementmedium

A state licensing agency runs an assistant that lets applicants check the progress of a permit. It calls a case record lookup and places the returned record in the prompt. In the last fortnight, 0.4 percent of sessions produced replies naming another applicant's permit details. In every affected trace, the other applicant's record is already present in the prompt input, and that applicant shares a postcode with the person asking. The integration team added a response cache to the case record lookup 16 days ago, and the model and prompt are unchanged. The agency must name the root cause before the service reopens. What is the most likely cause?

AThe model is carrying applicant details over from earlier sessions and recalling them when a postcode matches
BThe system prompt lacks an instruction forbidding replies about other applicants, so the model volunteers their data
CThe model is inventing plausible permit details from the postcode, which happen to match a real applicant's case
DThe new cache is keyed too coarsely, so a lookup can return a record stored for another applicant at the same postcode Correct

If leaked or wrong data is already present in the model's input, the fault lies in the integration that assembled it, not in the model. Reading the trace settles which layer is at fault: the other applicant's record was in the prompt before generation, so the model reproduced input rather than recalling or inventing it. That points to the code that fetches and inserts the record. The response cache is the one change on that path, the incidents began after it shipped, and the affected pairs share a postcode, which is the signature of a cache key that does not uniquely identify the applicant. Keying the cache on the authenticated applicant's identity, attached server-side from the session, removes the cause, and prompt rules or output filters are at most secondary safeguards.

Why A is wrong: Cross-session leakage feels like memory, so a model that remembers earlier users is an easy story to reach for. It is wrong because each request is processed from the input it is sent, with no recall of other sessions, and the trace already shows the other record arriving in the prompt from the lookup.

Why B is wrong: Adding a prompt rule is a quick change and would likely reduce how often the data is repeated, which makes it tempting. It is wrong as a root cause because the record should never have reached the prompt; an instruction is a compensating control that leaves another applicant's data flowing into every affected request.

Why C is wrong: Fabricated detail is a familiar model failure, so hallucination is a plausible first guess for unexpected content in a reply. It is wrong because the traces show the other applicant's actual record present in the input, so the model repeated real data it was given rather than generating it.

Why D is correct: The leaked record is in the prompt input, so it was fetched and inserted by the integration before the model ran. The cache is the one recent change on that path, the leaks began after it shipped, and the shared postcode suggests a cache key built from postcode or another field that does not identify a single applicant.

See more CCAR-P practice questions, answers explained.

Exam traps in Developer Productivity & Operational Enablement

Answers that look right on this material and are not. Each one is a distractor from a different question in the CCAR-P bank for this domain.

Rerun the failing chats on a larger model tier and check whether the wrong balances stop appearing
Why it is wrong: This is tempting because a model swap is a quick experiment and wrong numbers look like a model mistake. It is wrong because the only thing that changed was the account service, and a clean result on a larger tier would not show whether the bad figure arrived in the tool result, so it spends the single approved action without naming the layer.
In the model's reading of the tool result, because it turned a valid stock record into an out of stock reply
Why it is wrong: This is tempting because the customer-facing error is the model's reply, so the model looks like the component that got it wrong. It is wrong because the trace shows the tool result already read available: null when it reached the model, so the model reported faithfully on the input it was given rather than misreading a valid record.
Log the complete request payload, including the diff text, for every review during a two-week window
Why it is wrong: This is tempting because the full payload would answer the question completely. It is wrong because it writes source code into the logging system, which the security policy forbids, and metadata about the request is enough to detect truncation.

Examworthy is not affiliated with or endorsed by Anthropic. Original, blueprint-aligned practice material only.