NCA-GENL - Software Development (24% of the exam) - Section 2.3
Implement transformer-based models for different NLP applications.
Implement transformer-based models for NLP tasks including text classification, summarisation, question answering, and machine translation, selecting the correct pre-trained checkpoint and tokeniser for each. Recognise the differences between encoder-only, decoder-only, and encoder-decoder variants, and apply the right architecture to the task at hand.
Practice question for this objective
lock_openFree sampleSoftware Developmentmedium
A team must select a transformer architecture family for two separate applications: a sentiment classifier that produces one label per document, and an autoregressive assistant that generates free-form text token by token. They want each application matched to the architecture whose attention pattern suits it. Which architecture-to-task pairings are correct? Select TWO.
AAn encoder-only model with bidirectional self-attention fits the single-label document classifier, since each token can attend to the full left and right context before pooling.check_circle Correct
BA decoder-only model with causal masked self-attention fits the autoregressive assistant, since each position attends only to earlier tokens to predict the next one.check_circle Correct
CAn encoder-only model with causal masking is the natural fit for free-form text generation because masking future tokens is unique to encoder architectures.
DA decoder-only model with full bidirectional attention over the entire sequence is the correct choice for single-label document classification.
EBoth applications are best served by an encoder-decoder model because the cross-attention bridge is mandatory for any task that emits a class label or a token.
Bidirectional encoder-only models suit single-label classification while causal decoder-only models suit autoregressive next-token generation. Encoder-only transformers attend bidirectionally so each token sees the full context, giving representations ideal for a pooled classification head, whereas decoder-only transformers apply causal masking so each position depends only on prior tokens, which is precisely the constraint that next-token generation needs.
Why A is correct: Correct because bidirectional attention lets every token see the whole document, producing rich representations well suited to a single-label classification head.
Why B is correct: Correct because causal masking prevents a position from seeing future tokens, which is exactly what next-token generation requires.
Why C is wrong: Tempting by mixing terms, but wrong: encoders are bidirectional by default, and causal masking for generation is the hallmark of decoder-only models.
Why D is wrong: Wrong on two counts: decoder-only models use causal not bidirectional attention, and they are not the conventional pick for single-label classification.
Why E is wrong: Plausible since encoder-decoders are versatile, but wrong: cross-attention suits sequence-to-sequence mapping, not single-label classification or pure next-token generation.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENL bank for this domain.
The backbone must be re-pre-trained from scratch for each application so its internal representations specialise to that task's objective before any head is attached.
errorWhy it is wrong: Re-pre-training per task is tempting because each task differs, but it discards the whole point of transfer learning and is rarely necessary or affordable.
Use an encoder-only model for the translator, since its bidirectional attention can emit the target sentence one token at a time.
errorWhy it is wrong: Tempting because encoders are strong, but encoder-only models have no autoregressive decoder, so they cannot generate a target sequence token by token.
The encoder-only model attends only to the single preceding token at each position, while the decoder-only model attends to the entire sequence at once.
errorWhy it is wrong: This inverts the truth and also misstates the encoder, which makes it a trap for candidates who confuse the two, but encoders see the whole sequence and decoders are not limited to one prior token, so it is wrong.