Eval, Testing, and Debugging

An expenses assistant reads a tool that returns a supplier record. A platform team renamed the result field holding the supplier's tax identifier, and since that release reviewers have logged answers stating that no tax identifier is held for suppliers that plainly have one. The tool handler logs the raw payload it returns. A logged payload and the tool's declared result contract are reproduced below. Which diagnosis fits, and what is the correct first correction?

declared result contract: { "supplier": "string", "tax_id": "string" }
logged payload:          { "supplier": "Bramley Ltd", "tax_ref": "GB4471902" }

• AThe model misread a correct payload, so the system prompt should instruct it to check every field of a supplier record before stating that a value is absent from the record.
• BThe payload no longer matches the declared result contract, so the handler should validate the payload against that contract and fail the call loudly instead of returning an unrecognised shape.check_circle Correct
• CThe record was too large for the working context, so the handler should trim the payload to the fields the assistant reasons over before the result is appended to the conversation.
• DThe upstream release was faulty, so the assistant repository should pin the previous version of the supplier service and leave the tool result contract exactly as it stands today.

Why A is wrong: Tempting because instructions can improve careful reading, but the declared field genuinely is not in the payload, so no amount of instruction can recover a value the model was never sent.

Why B is correct: Correct: a schema mismatch at the integration boundary is the recorded cause, and validating the result turns a silent renamed field into an explicit failure the team can see and fix.

Why C is wrong: Tempting because pruning oversized tool output is a genuine technique, but the logged payload is two fields long and pruning would discard the renamed value rather than surface the mismatch.

Why D is wrong: Tempting because pinning does stop the bleeding for one consumer, but it leaves the handler unable to detect the next shape change and defers rather than removes the silent failure.

An incident assistant runs an agentic loop that calls a metrics tool, a log tool and a deployment history tool. One dependency intermittently accepts a connection and then never responds. The tool handler has no timeout, so the loop stops mid run and the user receives nothing at all, in roughly two per cent of requests. The team requires that a single unresponsive dependency never stall a run, and that the assistant may still answer from what it did retrieve provided the gap is stated. Which handling of the tool call meets both requirements?

• ALeave the tool handler unbounded but cap the number of loop turns, so a run that hangs is eventually ended by the turn limit and the user receives the partial answer produced so far.
• BApply a timeout to the tool call and, when it expires, abort the whole run and return a generic failure message, because an answer built on incomplete data cannot be trusted.
• CApply a timeout to the tool call and, when it expires, retry the same call repeatedly inside the same turn until it answers, so the loop continues with complete data.
• DApply a timeout to the tool call and, when it expires, return a tool result to the model marking that call as failed, letting the loop continue and requiring the answer to name the dependency it could not reach.check_circle Correct

Why A is wrong: Tempting because a turn cap is a sound loop safeguard, but the turn counter only advances when a turn completes, so a call that never returns never reaches the cap and the run still stalls.

Why B is wrong: Tempting because it is the conservative reading of correctness, but it discards the two successful lookups the run already holds, and the requirement explicitly permits a stated partial answer.

Why C is wrong: Tempting because it preserves completeness, but a dependency that accepts connections and never replies will absorb every retry, so the run stalls exactly as before with a longer wall-clock time.

Why D is correct: Correct because the timeout bounds the stall and the failed result is fed back as part of the conversation, so the loop proceeds with a model that knows one source is absent and can qualify its answer.

A team investigates a reported quality drop in a triage assistant by rerunning twenty tickets by hand. Between the two runs an engineer had also edited the system prompt, pinned a different model version and raised the output ceiling. The team's stated requirement is that any future quality claim be a measurement a reviewer can accept rather than an anecdote. What must the eval harness do to satisfy that requirement?

• ARun a fixed, version-controlled case set with agreed expected outcomes, hold the prompt, model version and request settings constant, and vary exactly one of them per run while recording the values used.check_circle Correct
• BRerun the same twenty tickets after every change and record a pass count, accepting whichever configuration was live at the time as the baseline for that week's comparison.
• CSample twenty fresh tickets from the previous day's live traffic on each run so the evaluation stays representative of the questions users are currently asking the assistant.
• DHave a second reviewer read the same twenty answers independently and treat agreement between the two reviewers as the accuracy figure reported to the business at each release.

Why A is correct: Correct: a fixed case set plus a recorded, single-variable configuration is what turns a score difference into evidence about the change rather than about three simultaneous edits.

Why B is wrong: Tempting because it is repeatable and cheap, but letting the live configuration drift into the baseline means the two numbers were produced under different conditions and cannot be compared.

Why C is wrong: Tempting because freshness guards against a stale case set, but resampling the inputs each run means the case set and the configuration both moved, so a score change has no attributable cause.

Why D is wrong: Tempting because independent review does reduce one reader's bias, but reviewer agreement measures the graders rather than the system, and the confounded configuration remains unaddressed.