Context Management & Reliability

Northgate's agent closes each session by stating a confidence percentage for its own resolution, and the Resolution Desk routes anything below 70 percent to a human. Reviewers find the stated figure runs high on billing disputes where get_customer returned no matching record, and low on straightforward returns the agent resolved correctly. An architect is asked what that number actually measures before the routing rule is written into the runbook. What does it measure?

• AText generated in the same way as the rest of the reply and conditioned on the conversation so far, which tracks how fluent and well supported the answer appears rather than whether the case was within the agent's competence.check_circle Correct
• BThe model's internal token probabilities for the answer it produced, surfaced as a percentage, which makes it a dependable proxy for correctness once the threshold has been tuned on real sessions.
• CThe proportion of the tools invoked during the session that returned a result without an error flag, which is the reason sessions holding an empty customer record still come out with a high score.
• DThe agreement between the agent's drafted reply and the policy text supplied in the system prompt, which is the reason contacts with no covering policy are scored as though they were straightforward.

Why A is correct: Correct: it is a prediction of what a plausible reply looks like, which is why an unsupported case with a tidy narrative reads as confident while a solved routine case can read as hedged.

Why B is wrong: Tempting because the figure looks like a probability and models do hold token level distributions, but a number the model writes into its reply is generated text, not a read out of those internals.

Why C is wrong: Tempting because an empty result does return cleanly and would inflate such a ratio, but nothing computes that ratio here; the model is writing a number rather than aggregating tool outcomes.

Why D is wrong: Tempting because an agreement check would be a sensible signal to build, but no such comparison is running, and a missing policy would leave that check with nothing to match against rather than a high score.

The escalation rules for the Customer Support Resolution Agent are written in terms of how hard a case looks: the system prompt tells it to hand over when a contact involves several steps or looks complicated. A quarterly review of 210 escalations finds that 61 per cent were multi-step but routine, such as an address correction followed by a reshipment, and the agent had already completed them correctly before handing over. Separately, 18 contacts asking for a goodwill credit, a remedy the published policy does not cover, were all single-step and were resolved by the agent inventing terms of its own. What should the architect do?

• ARaise the difficulty threshold so that a contact escalates only once it has consumed more than four tool calls, and log the tool-call count on every ticket so the team can tune the number against the next quarter's escalation figures.
• BRedefine the escalation triggers as boundary conditions on authority and policy coverage, namely a requested remedy with no published policy, an amount above the authorised limit, or a required record the tools cannot return, and remove the difficulty wording so that step count no longer drives handover.check_circle Correct
• CAdd a planning step in which the agent drafts the full sequence of actions it intends to take before acting on anything, and hand the contact to a person whenever that draft plan runs to more than three distinct steps.
• DHave the agent state a confidence score for its proposed resolution at the end of each contact, and route the contact to a human whenever that score falls below an agreed threshold set from the quarterly review sample.

Why A is wrong: Tempting because the count is objective and easy to tune, but step count is still a proxy for difficulty rather than for resolvability, so routine multi-step work keeps triggering handovers while the single-step goodwill credits stay well under any threshold.

Why B is correct: Correct because it separates the two populations the review exposed: routine multi-step work stays with the agent because nothing about it exceeds its authority, and the goodwill credits escalate because no published policy covers the remedy.

Why C is wrong: Tempting because an explicit plan makes the agent's intent inspectable before any tool runs, but the plan is then measured by its length, which is the same difficulty proxy in a new form and once again cannot flag a one-step request that policy does not permit.

Why D is wrong: Tempting because low confidence sounds like the signal a handover should follow, but a self-reported score tracks how familiar the wording feels rather than whether the remedy is permitted, and the invented goodwill terms would be reported with high confidence.

A claims processor extracts policy number, incident date and claimed amount from submitted forms and routes any field whose calibrated confidence falls below a set threshold into a human review queue, where every queued field is checked. Over eight weeks the proportion of queued fields that reviewers corrected fell from 9 percent to 3 percent, and the quality lead intends to report this as the pipeline's error rate falling by two thirds. Around 11 percent of fields enter the queue and the remainder are auto-approved and never inspected. What does the reviewer-corrected figure actually measure?

• AIt measures overall field accuracy for the pipeline, because any field with a realistic chance of being wrong is placed in the queue by construction and the auto-approved remainder therefore contributes no material share of the errors.
• BIt measures accuracy on the hardest fields the pipeline handles, and because those fields bound the error the auto-approved remainder can safely be assumed to be wrong no more often than the 3 percent observed in the queue.
• CIt measures the error rate inside the low-confidence stratum alone and says nothing about the 89 percent that was auto-approved, so estimating the errors that were never routed requires a stratified random sample of auto-approved fields checked against ground truth.check_circle Correct
• DIt measures the precision of the routing threshold and extrapolates cleanly to the whole population, because the queue is drawn from the same daily traffic as the auto-approved fields and shares its distribution of document types.

Why A is wrong: Tempting because the threshold is designed to select risky fields, so it feels exhaustive by intent. A threshold is a filter with an unmeasured miss rate, and treating its output as complete assumes the very property the queue was never instrumented to demonstrate.

Why B is wrong: The bounding argument sounds rigorous, which is what makes it dangerous. It holds only if confidence ranks difficulty correctly for every field, and an escaped error is by definition a hard field that scored high, so the assumption fails precisely on the cases that matter.

Why C is correct: Correct. The queue is a biased sample selected on confidence, so a statistic computed inside it describes only that stratum. Errors that scored above the threshold are absent from the measurement entirely, and only sampling the auto-approved stratum can estimate them.

Why D is wrong: Sharing a traffic source feels like a randomness argument, and that is the trap. Selection on confidence is what breaks the extrapolation rather than the source of the documents, so a common origin does not make the queue a representative sample.

A multi-agent research system produces reports in which every claim carries a citation back to a source. In an audit of one month of output, reviewers find that a typical run plans to consult twelve sources but successfully reads seven, and the finished report reads as an authoritative answer with no indication that five sources were missed. Two reviewers signed off on conclusions that the missing filings would have contradicted. Which change most effectively addresses the reviewers' problem?

• AHave the coordinator abandon any run in which a planned source could not be read, so that a reviewer is never handed a report that rests on an incomplete evidence base in the first place.
• BHave the report generation subagent close each report with a standing paragraph noting that automated research can be incomplete and that conclusions should be confirmed against the primary sources.
• CHave each subagent report the sources it planned to consult alongside the ones it actually read, and require the synthesis subagent to annotate each claim with the coverage behind it.check_circle Correct
• DHave the synthesis subagent grade its own confidence in each section of the draft and flag any section scoring below an agreed threshold for a second human review before the report is issued.

Why A is wrong: It is tempting because it guarantees that no partially sourced report reaches a reviewer. It is wrong because it throws away substantial completed work for one unreachable source, and a partial answer with its limits stated is more useful to a reviewer than no answer.

Why B is wrong: It is tempting because it is honest and takes one edit to ship. It is wrong because a blanket disclaimer applies equally to a fully sourced run and a badly degraded one, so it carries no information a reviewer can act on and is quickly ignored.

Why C is correct: Correct. Coverage is known at each subagent and lost at aggregation, so capturing planned against actual and carrying it into synthesis lets a reviewer see which specific claims rest on a thin evidence base and which are fully supported.

Why D is wrong: It is tempting because it targets weak sections rather than the whole report. It is wrong because self-reported confidence measures how coherent the retrieved material felt, not what was missing, so a section built on seven consistent sources scores high whatever the other five held.