A team maintains a fixed set of 80 test cases to compare prompt variants for a support summariser. When testing a new variant, an engineer adds 12 fresh cases that the new prompt handles well and removes 5 cases the old prompt passed but the new one fails. The new variant then reports a higher score. Why does this comparison fail to show whether the new prompt is genuinely better?
- AThe eval set is too small at 80 cases to detect any real difference between two prompt variants, so no conclusion can be drawn regardless of how the cases are chosen.
- BThe engineer should have switched to ROUGE scoring instead of pass or fail case counting, because overlap metrics are the only valid way to rank summarisation prompts.
- CThe new prompt was given more test cases than the old one, and a prompt evaluated on more examples will always achieve a higher average score than one evaluated on fewer.
- DChanging the test cases between the two runs means the variants were measured against different inputs, so the higher score reflects an easier test set rather than a better prompt. Correct
Why A is wrong: Tempting because small samples do weaken statistical power, but 80 cases can still surface a real difference; the decisive flaw here is that the test set itself was altered between variants, not its size.
Why B is wrong: Tempting because ROUGE is a common summarisation metric, but swapping the metric would not fix a benchmark that was changed between runs; the comparison is invalid for reasons unrelated to metric choice.
Why C is wrong: Tempting because the counts did change, but more examples do not mechanically raise an average score; the real problem is that the specific cases differed between the two runs, biasing the result.
Why D is correct: Correct: a fair comparison requires both prompts to face the identical eval set; selectively adding favourable cases and dropping failing ones changes the benchmark, so the score gain is an artefact of the new inputs.