AIF-C01 - Applications of foundation models (28% of the exam) - Section 3.3
Describe the training and fine-tuning process for foundation models.
Distinguish pre-training, fine-tuning, and instruction tuning, and understand the data and cost implications of each. Recognise when fine-tuning is justified and when prompting or retrieval is enough.
Pre-trainingFine-tuningInstruction tuning
Practice question for this objective
lock_openFree sampleApplications of foundation modelsmedium
A product team runs a customer assistant on a fluent general-purpose foundation model and is deciding whether to invest in fine-tuning. Prompt engineering and retrieval are already on the table as cheaper alternatives. Which two situations most strongly justify the added data and compute cost of fine-tuning rather than relying on prompting or retrieval alone? Select TWO.
AThe reference facts the assistant cites change several times each week and must always be current.
BA few thousand labelled in-house examples of the desired behaviour exist that can be used to adapt the model.check_circle Correct
CThe team simply wants to feed a few worked examples into the prompt to steer answers at request time.
DThe assistant must consistently adopt a specialised output style that prompting alone fails to hold across varied inputs.check_circle Correct
EThe model needs occasional access to a single internal document that a person could paste into the request.
Fine-tuning is justified when a labelled dataset exists and a durable behaviour or style is needed, whereas changing facts or one-off context favour retrieval or prompting. Fine-tuning pays off when the goal is a stable change in how the model behaves and there is labelled data to teach it. A specialised, consistent output style that prompting cannot hold, backed by a few thousand in-house examples, is exactly that case. By contrast, knowledge that changes weekly belongs in a retrieval store because retraining to refresh it is slow and costly, and steering with a handful of prompt examples or pasting a single document are jobs for prompting, not weight updates. The distractors each describe a need that a cheaper method already covers.
Why A is wrong: Frequently changing facts favour retrieval, since fine-tuning bakes knowledge into weights that go stale the moment the data changes.
Why B is correct: Having a curated labelled dataset of the target behaviour is what makes fine-tuning feasible and worthwhile over prompting.
Why C is wrong: Supplying examples in the prompt is few-shot prompting, a lighter approach that avoids fine-tuning's data and compute cost entirely.
Why D is correct: A durable behaviour or style the base model will not reliably reproduce from instructions is a classic case where fine-tuning earns its cost.
Why E is wrong: Pasting one document into context is trivially handled by prompting or retrieval, so it gives no reason to fine-tune anything.
Answers that look right on this material and are not. Each one is a distractor from a different question in the AIF-C01 bank for this domain.
Continued pre-training on a large corpus of unlabelled domain text
errorWhy it is wrong: Continued pre-training extends the model's general knowledge but does not teach it to respond to explicit instructions; it does not optimise instruction-following behaviour specifically.
Continued pre-training on a curated subset of high-quality web text
errorWhy it is wrong: Continued pre-training extends the model's general knowledge but does not use human preference signals; it cannot target the alignment with user preferences described in the scenario.
Raw web-scraped text with no annotations, shuffled into random batches
errorWhy it is wrong: Raw unannotated text is suitable for pre-training self-supervised objectives, not for instruction tuning, which requires explicit (instruction, response) pairings to teach the model the desired behaviour.