ExamworthyExamworthy
Open the appStart free

NCA-GENL - Experimentation - Section 3.4

Test and compare model performance on question-answering tasks.

Design evaluation experiments that measure model performance on question-answering benchmarks, using metrics such as Exact Match and F1 to compare extractive and generative approaches. Distinguish between open-domain and closed-domain QA settings, and identify factors such as context length and retrieval quality that affect accuracy.

Practice question for this objective

Free sampleExperimentationmedium

A team evaluates two extractive QA models on a SQuAD-style dataset. Model A achieves an exact match (EM) of 68% and an F1 score of 81%. Model B achieves an EM of 72% and an F1 score of 74%. Which conclusion is most defensible when reporting these results?

  • AModel B is strictly superior because its exact match score is higher on the held-out test set.
  • BThe two metrics capture different aspects of answer quality, so the preferred model depends on whether partial credit is acceptable in the target application. Correct
  • CModel A is strictly superior because its F1 score is higher on the held-out test set.
  • DThe evaluation is invalid because SQuAD-style metrics cannot compare extractive models that differ in architecture.
Distinguish when exact match versus F1 is the appropriate metric for evaluating and comparing extractive QA model outputs. On SQuAD-style benchmarks, exact match awards a point only when the predicted answer string matches a reference answer exactly, making it a strict measure. F1 computes token-level overlap between prediction and reference, rewarding partial answers. Because the two metrics measure different things, a model can lead on one while trailing on the other. The correct comparison strategy is to report both, understand the application tolerance for partial answers, and select the preferred model accordingly rather than declaring one universally superior.

Why A is wrong: Tempting because exact match is the stricter metric and Model B leads there. Wrong because EM penalises any token mismatch, so a model that recovers most of the answer but misses a stop-word scores zero; Model A's higher F1 shows it retrieves more answer tokens on average, making the trade-off non-trivial.

Why B is correct: Exact match requires the predicted span to match the reference exactly, while F1 measures token-level overlap and awards partial credit. Choosing between them is an application decision: high-stakes retrieval may require exact match; conversational assistants may tolerate partial matches. Reporting both and contextualising the trade-off is the defensible approach.

Why C is wrong: Tempting because F1 is often cited as the primary SQuAD metric and Model A leads there. Wrong because neither metric dominates in all use cases; the right choice depends on whether partial credit matters to the application, so a one-dimensional superiority claim based on a single metric is unjustified.

Why D is wrong: Tempting because architectural differences can affect comparability in some evaluation frameworks. Wrong because SQuAD EM and F1 are output-level metrics applied to predicted answer spans regardless of model architecture; they are specifically designed to compare any extractive QA system on a shared held-out benchmark.

See more NCA-GENL practice questions, answers explained.

More in this domain

Back to all Experimentation objectives, or the NCA-GENL cert hub.

Examworthy is not affiliated with or endorsed by NVIDIA. Original, blueprint-aligned practice material only.