NCA-GENL - Experimentation (22% of the exam) - Section 3.3
Experiment with transformer-based models for various NLP tasks.
Experiment with pre-trained transformer models by varying tokenisation strategies, prompt formats, and fine-tuning hyperparameters across NLP tasks such as sentiment analysis, named-entity recognition, and text generation. Interpret evaluation metrics such as F1 score, BLEU, and perplexity to guide further iterations.
Practice question for this objective
lock_openFree sampleExperimentationmedium
A research team is planning experiments across two NLP tasks: a free-form abstractive summarisation task that must generate fluent novel sentences, and a token-level named-entity recognition task that labels each input token. They want to choose a transformer architecture family that naturally suits each task before fine-tuning. Which two pairings of architecture to task are most appropriate for these experiments? Select TWO.
AUse a decoder-only autoregressive transformer for the abstractive summarisation task, because it generates output tokens one at a time conditioned on prior tokens.check_circle Correct
BUse an encoder-only bidirectional transformer for the token-level named-entity recognition task, because it produces contextual per-token representations for classification.check_circle Correct
CUse an encoder-only bidirectional transformer for the abstractive summarisation task, because its masked-language objective writes fluent multi-sentence summaries directly.
DUse a decoder-only autoregressive transformer for the named-entity recognition task, because its causal masking gives each token full bidirectional context for labelling.
EUse a convolutional sequence model for both tasks, because fixed-width filters capture all long-range dependencies that transformer attention captures.
Match transformer architecture families to NLP task types, pairing autoregressive decoders with generation and bidirectional encoders with per-token labelling. Architecture choice follows the task shape: generation needs left-to-right conditioning that an autoregressive decoder provides, while per-token labelling needs the full bidirectional context that an encoder produces. Causal masking and the absence of a decoder are what disqualify the wrong pairings.
Why A is correct: Decoder-only autoregressive models are built for left-to-right generation, which matches producing novel summary sentences token by token.
Why B is correct: Encoder-only bidirectional models give every token a context-rich representation, ideal for per-token labelling such as entity recognition.
Why C is wrong: Tempting since encoders understand text well, but encoder-only models lack an autoregressive decoder and cannot freely generate novel summary sentences.
Why D is wrong: Plausible because decoders are powerful, but causal masking hides future tokens, so it does not give the full bidirectional context entity labelling benefits from.
Why E is wrong: Convolutional filters have a limited receptive field and do not capture long-range dependencies as attention does, so this misrepresents the trade-off.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENL bank for this domain.
Perplexity computed on a held-out portion of the source documents
errorWhy it is wrong: Perplexity measures how well a language model predicts a held-out text and reflects fluency, not content coverage. It does not compare a generated summary against a reference summary and so cannot assess whether key facts were retained.
The prompt template performs the network call to the model provider and returns the generated tokens, so no separate model component is required in the chain.
errorWhy it is wrong: Tempting because the template precedes the model, but it only formats inputs; the model component, not the template, performs the generation call.
Encoder-only models generate text autoregressively one token at a time, whereas decoder-only models produce a single fixed-length embedding for the entire input.
errorWhy it is wrong: Tempting because both architectures do produce embeddings, but the roles are reversed: decoder-only models are the autoregressive generators and encoder-only models produce contextual per-token representations rather than generating text.