Prompt engineering and evaluation

A developer is testing a new prompt intended to make a model translate informal customer feedback into formal written summaries. The first attempt produces summaries that retain colloquial phrasing. Which prompt engineering adjustment should the developer try first?

• ADecrease the model's top-p sampling parameter to produce more deterministic output.
• BAdd a few-shot example pairing one informal feedback excerpt with its correct formal summary before the live input.check_circle Correct
• CRewrite the prompt to ask the model to summarise the feedback without mentioning formality.
• DIncrease the maximum output token limit so the model has room to restate the feedback in full before summarising.

Why A is wrong: Lowering top-p makes output more predictable but does not instruct the model on what constitutes formal phrasing; the model can deterministically produce informal language if that matches its priors for this input.

Why B is correct: A few-shot example directly demonstrates the mapping from informal to formal register, giving the model a concrete reference for what formalisation looks like, which is more targeted than any sampling parameter adjustment.

Why C is wrong: Removing the formality instruction would make the problem worse; without guidance on register, the model has no reason to shift away from informal phrasing.

Why D is wrong: More output tokens do not address the formalisation problem; a longer informal summary is still informal. The issue is the model's understanding of the target register, not the output length.

A team iterates on a prompt for sentiment analysis and achieves 94% accuracy on their internal eval set. When deployed, real-world accuracy drops to 78%. Which factor is the most likely explanation?

• AThe foundation model was not fine-tuned on sentiment data and therefore cannot generalise.
• BThe eval set was not representative of the production input distribution, so the prompt was over-tuned to the eval data.check_circle Correct
• CThe inference latency in production is higher than in evaluation, causing the model to time out more frequently.
• DThe team used a different foundation model version in evaluation than in production.

Why A is wrong: A 94% score on the eval set shows the model can perform sentiment analysis adequately. The issue is that this accuracy does not transfer to the real world, which is a data distribution problem rather than a model capability problem.

Why B is correct: When the eval set does not match the real-world query distribution, prompt iteration converges on characteristics of the eval data rather than on the actual task. The model appears to perform well in evaluation but the high score reflects fitting to a non-representative sample. This is the prompt-evaluation equivalent of overfitting.

Why C is wrong: Timeouts would cause request failures, not a drop in sentiment accuracy. The gap between eval and production accuracy is a quality issue, not an availability or latency issue.

Why D is wrong: A model version mismatch is possible but is not indicated by the scenario. The most parsimonious explanation for a systematic eval-to-production gap is a distributional mismatch in the eval set, not an infrastructure discrepancy.

A developer adds detailed output formatting instructions to a prompt and observes that factual accuracy on the eval set improves from 81% to 88%. What is the correct interpretation of this result before accepting the change?

• AThe formatting instructions directly caused the accuracy improvement, so the change should be accepted immediately.
• BThe change also improved format compliance, which proves the formatting instructions are correct.
• CThe improvement may be partially caused by random variation; a significance test on the eval set size is needed to judge confidence.check_circle Correct
• DAn 88% accuracy score means the prompt has reached production-ready quality regardless of the task requirements.

Why A is wrong: A single observed improvement does not confirm causation. Other factors - such as a concurrent change to the eval set, model version drift, or random variation - could account for part or all of the difference. Causation requires controlled conditions.

Why B is wrong: Format compliance and factual accuracy are separate metrics. Improved format compliance is expected when formatting instructions are added; it does not validate the accuracy improvement or confirm that the overall change is safe to deploy.

Why C is correct: Even a real improvement must be distinguished from sampling noise. The appropriate response is to assess statistical significance - for example, using a confidence interval or a proportion test given the eval set size - before concluding the change reliably improves accuracy. A 7-percentage-point gain on a small eval set may not be significant.

Why D is wrong: What counts as production-ready depends on the task domain and risk tolerance, not an absolute threshold. For safety-critical applications, 88% may be far too low. The change must also be tested for regressions on edge-case subsets before deployment.

During prompt evaluation for a medical triage chatbot, human raters disagree on whether a model output is 'clinically appropriate'. What metric should be calculated first to assess the reliability of the human evaluation process itself?

• APrecision and recall across the clinical appropriateness labels assigned by each rater separately.
• BThe F1 score of the chatbot relative to the raters' majority-vote labels.
• CInter-rater agreement (for example, Cohen's kappa) across all rater pairs on the shared sample.check_circle Correct
• DThe BLEU score of the chatbot outputs against a set of reference responses written by a senior clinician.

Why A is wrong: Precision and recall measure how well a classifier matches a ground-truth reference. Before computing those, the team needs to know whether the raters agree on what the ground truth is - an inter-rater reliability problem, not a precision/recall problem.

Why B is wrong: F1 tells you how the chatbot performs against a reference label set. If the raters themselves disagree heavily, the majority-vote label is unreliable, and the resulting F1 score is misleading. Inter-rater agreement must be established first.

Why C is correct: Inter-rater reliability measures the degree to which independent raters assign the same label to the same output. Low agreement means the evaluation rubric is ambiguous or the criteria are insufficiently defined. Fixing this is a prerequisite before trusting any aggregated quality scores.

Why D is wrong: BLEU measures n-gram overlap, which is suited to machine translation rather than open-ended clinical text quality. It also does not address the rater disagreement problem identified in the stem.

A team uses few-shot prompting to classify product reviews as positive, neutral, or negative. They include eight examples in the prompt, but six of them are positive reviews and two are negative, with no neutral example. In testing, the model rarely returns the neutral label even for clearly neutral reviews. Which property of the example set best explains this skew?

• AThe number of examples exceeds the few-shot limit, so the model discards the later demonstrations and only the first positive examples influence its output.
• BThe demonstrations are imbalanced across the label classes and omit the neutral category, so the model learns a distribution that under-represents the labels it was rarely or never shown.check_circle Correct
• CThe examples are written in inconsistent tenses, which prevents the model from forming a stable mapping between review text and sentiment label.
• DThe examples should have been placed after the review to be classified rather than before it, which reversed the order the model reads them in.

Why A is wrong: Tempting because long prompts can lose detail, but there is no fixed few-shot count that the model silently truncates; eight short examples are well within normal limits, so discarding is not the cause.

Why B is correct: Correct: few-shot examples shape the output distribution the model infers, so an imbalanced set that never shows a neutral case biases predictions away from that under-represented class.

Why C is wrong: Plausible because parallel phrasing helps, but minor tense variation does not block sentiment mapping; the visible skew toward positive tracks the example distribution, not grammar.

Why D is wrong: Tempting because ordering can matter, but putting demonstrations before the target input is standard and correct; the missing neutral class, not example position, drives the skew.