Pure self-attention computes each output as a weighted sum over all input tokens, with weights derived from content-based query-key similarity. A practitioner notes that, given only token embeddings, shuffling the order of the input tokens would leave each token's attention output unchanged. Which property does this observation reveal, and what does a transformer add to overcome it?
- ASelf-attention is invariant to embedding magnitude, so transformers add layer normalisation to restore the model's sensitivity to the scale of each individual token vector.
- BSelf-attention is permutation-equivariant to token order, so transformers add positional encodings to the token embeddings to inject sequence-order information. Correct
- CSelf-attention is non-linear in its input, so transformers add a feed-forward sub-layer to make the mapping from tokens to outputs injective and therefore aware of order.
- DSelf-attention is sparse over long sequences, so transformers add a causal mask to ensure each token can recover its own absolute position directly from the masking pattern.
Why A is wrong: Tempting because layer normalisation is a real transformer component, but the observation concerns order rather than magnitude, and normalisation standardises feature scale rather than supplying any positional information.
Why B is correct: Correct: because attention weights depend only on content similarity, reordering tokens permutes the outputs identically and the operation carries no inherent notion of position. Transformers therefore add positional encodings to the input embeddings so that order information is available to the model.
Why C is wrong: Tempting because the feed-forward sub-layer does add non-linearity, but it is applied identically per position and adds no order information; the shuffling invariance described is about position, which the feed-forward sub-layer cannot resolve.
Why D is wrong: Tempting because a causal mask does break the symmetry between earlier and later tokens, but standard attention is dense rather than sparse, and the mask alone does not encode absolute position; explicit positional encodings are the mechanism transformers use.