A legal drafting service sends a large unchanging clause library at the head of every request, followed by the lawyer's short question. The team enables prompt caching to meet a stated per-request cost ceiling, but the request builder writes a generated line carrying the current timestamp and the requesting user's name immediately above the clause library. Spend per request does not fall, and an engineer concludes that caching only ever helps latency. How should the situation be read?
- AThe engineer is right that caching is a latency feature, so the ceiling should instead be met by cutting the output limit on every request until spend per request falls inside it.
- BCaching applies to generated replies rather than to sent input, so it can only help when two lawyers ask a question worded identically, which this traffic does not contain.
- CCaching cannot apply while the clause library is this large, so the remedy is to move the service to a smaller model and accept whatever drop in drafting quality that brings.
- DCaching is primarily a cost control, and it is being defeated because the timestamp and user name change the beginning of every request; move that volatile text below the clause library so the prefix is identical on each call. Correct
Why A is wrong: Tempting because caching does improve time to first token and the output limit is easy to change. It is wrong on both counts: caching is primarily a cost control on repeated input, and a lower output limit truncates drafted clauses rather than reducing the repeated input that dominates the bill.
Why B is wrong: Tempting because a response cache keyed on identical questions is a familiar pattern from ordinary web services. It is wrong because prompt caching operates on the repeated portion of the request sent to the model, not on the replies that come back.
Why C is wrong: Tempting because a smaller model is the standard lever when a per-request ceiling is missed. It is wrong because size is not what defeated the cache, and downsizing without measuring the drafting quality bar trades a solvable configuration fault for a quality regression.
Why D is correct: Correct because a cache serves the longest identical leading portion of a request, and volatile text placed above the library makes every request differ from the first token onwards.