NCA-GENM - Core Machine Learning and AI Knowledge (20% of the exam) - Section 2.2
Describe how transformers function as foundational building blocks for modern LLMs.
Describe the encoder-decoder structure of transformers, including self-attention, feed-forward layers, and positional encodings. Explain how scaling these components produces the emergent capabilities seen in large language models.
Transformers
Practice question for this objective
lock_openFree sampleCore Machine Learning and AI Knowledgemedium
Pure self-attention computes each output as a weighted sum over all input tokens, with weights derived from content-based query-key similarity. A practitioner notes that, given only token embeddings, shuffling the order of the input tokens would leave each token's attention output unchanged. Which property does this observation reveal, and what does a transformer add to overcome it?
ASelf-attention is invariant to embedding magnitude, so transformers add layer normalisation to restore the model's sensitivity to the scale of each individual token vector.
BSelf-attention is permutation-equivariant to token order, so transformers add positional encodings to the token embeddings to inject sequence-order information.check_circle Correct
CSelf-attention is non-linear in its input, so transformers add a feed-forward sub-layer to make the mapping from tokens to outputs injective and therefore aware of order.
DSelf-attention is sparse over long sequences, so transformers add a causal mask to ensure each token can recover its own absolute position directly from the masking pattern.
Self-attention is order-agnostic, so transformers add positional encodings to give the model sequence-position information. Attention scores depend solely on content-based query-key similarity, so permuting the input tokens permutes the outputs the same way and conveys no position. Without an explicit signal the model cannot distinguish word order. Positional encodings, added to the token embeddings at the input, supply this missing order information so attention can use position.
Why A is wrong: Tempting because layer normalisation is a real transformer component, but the observation concerns order rather than magnitude, and normalisation standardises feature scale rather than supplying any positional information.
Why B is correct: Correct: because attention weights depend only on content similarity, reordering tokens permutes the outputs identically and the operation carries no inherent notion of position. Transformers therefore add positional encodings to the input embeddings so that order information is available to the model.
Why C is wrong: Tempting because the feed-forward sub-layer does add non-linearity, but it is applied identically per position and adds no order information; the shuffling invariance described is about position, which the feed-forward sub-layer cannot resolve.
Why D is wrong: Tempting because a causal mask does break the symmetry between earlier and later tokens, but standard attention is dense rather than sparse, and the mask alone does not encode absolute position; explicit positional encodings are the mechanism transformers use.
Exam traps in Core Machine Learning and AI Knowledge
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENM bank for this domain.
Self-attention has a built-in left-to-right recurrence that was disrupted, so the encoder lost its ability to process tokens sequentially.
errorWhy it is wrong: This sounds plausible because order matters here, but self-attention has no recurrence at all and processes all tokens in parallel, so there is no sequential mechanism to disrupt.
Self-attention stores a learned lookup table indexed by token position, so adding new positional entries for image patches extends alignment without changing the key or query projections.
errorWhy it is wrong: Positional encodings inform the model of sequence order but do not control cross-modal compatibility. Attention scores are derived from query-key dot products, not from a positional lookup table, so this description misattributes the source of alignment.
The feed-forward sub-layers are skipped for most tokens once the context exceeds a few thousand positions, so only attention continues to consume resources at long lengths.
errorWhy it is wrong: This is tempting because feed-forward cost does look smaller in profiles, but the feed-forward layers run on every token at every position and are never skipped, so the claim about skipping is false.