AIF-C01 - Applications of foundation models (28% of the exam) - Section 3.4
Describe methods to evaluate foundation model performance.
Use human evaluation, benchmark datasets, and task-specific metrics to judge a model, and understand the limits of any single metric. Recognise why evaluation must reflect the real task, not a proxy.
BenchmarksHuman evaluationTask metrics
Practice question for this objective
lock_openFree sampleApplications of foundation modelshard
An evaluation lead is assembling a plan to assess a foundation model that will power a clinical-summary feature. The lead wants the plan to combine evaluation methods so no single number decides the verdict and the assessment mirrors the real task. Which two statements about choosing evaluation methods are correct? Select TWO.
AA public benchmark gives a comparable, repeatable score but can overstate true ability if its items already appeared in the model's training data.check_circle Correct
BA task-specific metric computed on a held-out set built from real clinical summaries tracks the deployed task more closely than a generic benchmark does.check_circle Correct
CA single automated accuracy figure on any held-out set is enough on its own to decide the verdict once it clears a chosen pass threshold.
DHuman expert review should be dropped from the plan here, because clinician judgements are slower and more costly than running automated scoring.
EReporting only the overall mean score across all cases is preferable to also inspecting performance on the rare but high-stakes summary types.
A sound evaluation plan combines benchmarks, task-specific metrics, and human review, because each method has limits and the test data should mirror the deployed task. Public benchmarks offer comparable, repeatable scores but can be inflated by training-data contamination, so they cannot stand alone. A task-specific metric measured on held-out data drawn from the real clinical summaries is a closer proxy for production performance. Relying on a single aggregate number, dropping expert review for cost reasons, or reporting only a mean hides failure modes and conflicts with the goal of an evaluation that mirrors the real task.
Why A is correct: Standardised benchmarks aid comparison yet are open to contamination, so a high score can reflect memorisation rather than genuine capability.
Why B is correct: Evaluating on data drawn from the actual task narrows the gap between test and production, making the result a more faithful proxy for live behaviour.
Why C is wrong: One aggregate number hides failure modes and task mismatch, which is why the plan should combine complementary methods rather than trust a threshold.
Why D is wrong: Cost and speed do not make expert review unsound; for clinical correctness it captures quality that automated overlap scores miss, so it stays in.
Why E is wrong: An overall mean can mask poor performance on rare critical cases, so slicing the results is needed rather than collapsing everything to one figure.
Answers that look right on this material and are not. Each one is a distractor from a different question in the AIF-C01 bank for this domain.
Pick Model J on the basis that one objective numeric score is inherently more trustworthy than the subjective agent ratings of the replies.
errorWhy it is wrong: An automated number is not automatically more valid; here it measures a proxy that conflicts with the task the agents judge, so it is weaker.
Perplexity, which measures how well the model's language distribution fits a held-out test corpus, providing a generation quality signal.
errorWhy it is wrong: Perplexity reflects how surprised the model is by a test corpus and correlates with fluency, but it does not compare generated output against reference summaries. A fluent but off-topic summary would have low perplexity yet poor ROUGE.
The three benchmarks use different scoring scales, making cross-benchmark comparison statistically invalid without normalisation.
errorWhy it is wrong: Different scales are a presentation issue and can be addressed by normalisation. This is a procedural concern, not the most direct reason why high benchmark scores might be misleading about generalisation.