NCA-GENL - Experimentation (22% of the exam) - Section 3.2
Leverage transfer learning between models.
Apply transfer learning by loading a pre-trained model and adapting its learned representations to a new domain or task with limited labelled examples. Compare strategies such as freezing backbone layers versus full fine-tuning, and evaluate the resulting accuracy and training cost trade-off.
Transfer learning
Practice question for this objective
lock_openFree sampleExperimentationmedium
When adapting a pre-trained transformer to a downstream task, practitioners distinguish feature extraction from full fine-tuning. Which statement most accurately contrasts these two transfer-learning strategies?
AFeature extraction keeps the pre-trained backbone frozen and trains only a new head, whereas full fine-tuning updates the backbone weights along with the head.check_circle Correct
BFeature extraction always produces higher downstream accuracy than full fine-tuning because frozen weights cannot drift away from their pre-trained values.
CFull fine-tuning trains only the embedding layer while leaving every transformer block frozen, which is why it needs less data than feature extraction.
DFeature extraction and full fine-tuning both update every weight in the network and differ only in the learning rate applied to the output head.
Feature extraction freezes the pre-trained backbone and trains only a new head, while full fine-tuning updates the backbone weights as well. In feature extraction the pre-trained layers act as a fixed encoder and gradients flow only into the newly added task head. Full fine-tuning removes that freeze so gradients also update the backbone, letting the shared representation specialise to the task at the cost of more data and compute.
Why A is correct: Correct. Feature extraction uses the frozen backbone as a fixed feature encoder and trains only the task head; full fine-tuning lets gradients update the backbone too, adapting the representation itself.
Why B is wrong: Tempting because frozen weights avoid catastrophic forgetting, but full fine-tuning usually reaches higher accuracy when enough labelled data is available, so 'always' is incorrect.
Why C is wrong: Tempting because partial-layer training exists, but full fine-tuning updates the whole network, not just embeddings, and it generally needs more data, not less.
Why D is wrong: Tempting because learning-rate tuning matters, but feature extraction by definition freezes the backbone, so the claim that both update every weight is wrong.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENL bank for this domain.
Whether the downstream training set is larger than the corpus the source model was pre-trained on, since transfer only helps when target data dominates.
errorWhy it is wrong: Tempting because data size matters, but transfer learning is most valuable precisely when target data is scarce relative to the pre-training corpus, so this reverses the real relationship.
Apply a single global learning rate to all layers and rely on early stopping to halt training before the lower layers drift too far.
errorWhy it is wrong: Tempting because early stopping does limit overall drift, but a single global rate updates every layer with the same step size, so it cannot make the lower layers change less than the upper layers as required.
Pre-training permanently freezes all weights, so the downstream task only learns a new output layer and never alters the shared representation.
errorWhy it is wrong: Tempting because feature extraction with frozen layers is one transfer strategy, but it is not the defining principle; full fine-tuning updates the shared weights, so freezing is optional, not inherent.