NCA-GENL - Software Development (24% of the exam) - Section 2.2
Leverage transfer learning for efficient results with less data and computation.
Use transfer learning to initialise a model with weights pre-trained on a large dataset, then fine-tune only the final layers on a smaller domain-specific dataset to reduce both training time and data requirements. Distinguish full fine-tuning from feature extraction, and recognise when each is preferable given available compute and labelled data.
Transfer learning
Practice question for this objective
lock_openFree sampleSoftware Developmentmedium
A team is building a text classification model to detect customer complaint categories. They have 800 labelled examples and limited GPU budget. Which approach best leverages transfer learning to achieve strong performance under these constraints?
ALoad a pre-trained language model, freeze the lower encoder layers, and fine-tune only the upper layers and classification head on the 800 examples.check_circle Correct
BTrain a transformer model from scratch on the 800 examples using the full parameter set and standard cross-entropy loss.
CApply data augmentation to expand the 800 examples to 80,000 synthetic samples, then train a transformer from scratch on the augmented set.
DUse a pre-trained model as a feature extractor with all layers frozen, pass its output embeddings to a small logistic regression head, and skip any gradient updates to the base model.
Identify how freezing pre-trained layers and fine-tuning only upper layers reduces data and compute requirements in transfer learning. Transfer learning achieves efficiency by reusing general-purpose representations learned during pre-training on large datasets. Freezing lower encoder layers prevents catastrophic forgetting of these representations and reduces the number of gradient updates needed, so the model converges on a small labelled set with a fraction of the compute required for training from scratch.
Why A is correct: Freezing lower layers preserves general linguistic representations already learned on large corpora, dramatically reducing the number of trainable parameters. Fine-tuning only the upper layers and head allows the model to adapt to the target domain with far less data and computation, which is the core efficiency benefit of transfer learning.
Why B is wrong: Tempting because training from scratch gives full control over the architecture, but with only 800 examples and limited compute, a randomly initialised model will underfit severely and waste the available GPU budget without the representational head-start that a pre-trained model provides.
Why C is wrong: Augmentation can help but does not replace the rich representations already encoded in a pre-trained model. Training from scratch on synthetic data still requires substantial compute and risks amplifying label noise, so this does not leverage transfer learning and defeats the efficiency objective.
Why D is wrong: Using frozen embeddings as features is a valid but weaker form of transfer learning. Because none of the pre-trained weights are updated to the target distribution, the model cannot adapt its representations to domain-specific vocabulary or phrasing, typically yielding lower accuracy than partial fine-tuning on the same data budget.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENL bank for this domain.
Fine-tuning permanently freezes every pre-trained weight, so no parameters change and the optimiser therefore needs almost no labelled data to fit the task.
errorWhy it is wrong: Tempting because freezing does cut cost, but standard fine-tuning updates weights; claiming none change describes feature extraction, not why fine-tuning works.
Re-initialise the pre-trained weights to random values before training so the optimiser is not biased by the source task and can fit the domain freely.
errorWhy it is wrong: Tempting as a way to avoid source bias, but discarding pre-trained weights throws away the transfer benefit and demands far more data and compute.
Fine-tuning freezes the entire network and trains only a fresh output layer, so the small labelled set never needs to update any internal representation.
errorWhy it is wrong: Freezing all but the head is one valid feature-extraction variant, but it is not the defining principle of transfer learning, and full or partial fine-tuning that updates internal layers is common, so this description is too narrow and wrong as a general rule.