A team improving a ticket triage prompt made four edits in one release: it added a block of worked examples, rewrote the role sentence, moved the operating rules from the user turn into the system prompt, and added an explicit output contract. Accuracy on a fixed labelled set of 200 tickets rose by six points, and cost per request rose by about a third, driven by the example block. Finance holds a per-request ceiling, and the team must establish which edits earn their cost before the release goes wider. How should the team proceed?
- AShip all four edits together and watch live accuracy for a month, on the basis that the combined result already clears the agreed accuracy bar for the assistant.
- BRemove the example block alone and re-run the labelled set, keeping the block out of the prompt if accuracy holds within one point of the combined result.
- CRe-run the labelled set with the edits applied one at a time from the original prompt, so the effect of each edit on accuracy and on cost is measured separately. Correct
- DAsk the assistant to review the four edits and explain which of them it found most influential on its own answers, then drop the edits it does not name.
Why A is wrong: Tempting because the combined measurement is real and the bar is met, but it keeps whatever share of the extra cost buys nothing, and live traffic gives no way to attribute the gain to any one edit.
Why B is wrong: Tempting because it isolates the expensive edit and would answer the cost question, but it leaves the other three edits unattributed, so a change that harms accuracy stays hidden inside the bundle.
Why C is correct: Correct because changing one element per run against a fixed scored set is what makes an effect attributable, and it prices each edit so a costly one that adds little accuracy can be dropped.
Why D is wrong: Tempting because the model can produce a fluent account of its own behaviour, but that account is generated text rather than a measurement, and the team already holds a scored set that settles the question.