A national tax agency's online assistant answers questions about a citizen's own filings through a lookup tool whose input includes a taxpayer identifier that the model fills in. In a red-team exercise, testers signed in as one citizen and claimed to be acting for a relative, and in 7 of 40 attempts the assistant retrieved the relative's filing. The agency requires that a signed-in citizen can retrieve their own records alone, and the assistant must keep answering filing questions without staff involvement. What should the architect recommend?
- AStrengthen the system prompt to say the assistant must look up the signed-in citizen's identifier and refuse others
- BScreen incoming messages with a classifier that blocks requests mentioning a relative, an agent or another person
- CHave the tool handler take the taxpayer identifier from the authenticated session and ignore any model-supplied value Correct
- DLog each lookup with both identifiers and have the fraud team review mismatched pairs in a weekly audit report
Why A is wrong: This is tempting because the failures came from the model being talked into a different identifier, and a firmer instruction would reduce that. It is wrong because the model still controls the identifier, so a sufficiently persuasive request can still succeed; an instruction is not an access control.
Why B is wrong: Input screening is a genuine layer and would catch the red team's phrasing. It is wrong because attackers can rephrase around the classifier, it blocks legitimate questions that mention family, and the identifier is still taken from model output.
Why C is correct: This is correct because the record scope is attached server-side from the session, so no wording in the conversation can change which taxpayer is looked up. The assistant keeps answering filing questions automatically, which meets the no-staff constraint.
Why D is wrong: Audit logging is valuable and an assessor would expect it. It is wrong as the answer because it detects a disclosure days after it has happened rather than preventing it, and it leaves the model in control of whose record is fetched.