A team is building an evaluation set to judge whether changes to a customer-support summarisation prompt genuinely improve quality rather than just helping on a few familiar cases. Which two characteristics make the evaluation set a trustworthy basis for deciding that a prompt change will generalise to unseen production inputs? Select TWO.
- AIt contains a varied spread of input types and difficult edge cases that mirror the distribution of real production traffic. Correct
- BIt is held fixed across every prompt variant being compared so a change in score reflects the prompt and not the data. Correct
- CIt is small enough for a reviewer to re-run by hand within a minute after each wording tweak to the prompt.
- DIt is limited to the inputs that the current production prompt already summarises correctly and cleanly today.
- EIt is regenerated from the newest variant's strongest sample outputs immediately before each comparison run.
Why A is correct: Representative, varied coverage is what lets a score on the set predict behaviour on unseen inputs rather than on a narrow slice.
Why B is correct: Holding the set constant isolates the prompt as the only variable, which is required for a fair like-for-like comparison.
Why C is wrong: Convenience and speed are practical but do not make the set representative, so a fast tiny set can still give a misleading signal.
Why D is wrong: Including only inputs the prompt already passes inflates the score and hides the failures iteration is meant to surface.
Why E is wrong: Rebuilding the set from a variant's own best outputs biases the test toward that variant and breaks comparability.