A researcher notices that a transformer language model produces identical output regardless of the order in which tokens appear in the input sequence. Which component is most likely missing or misconfigured?
- AThe feed-forward sublayer within each encoder block.
- BThe positional encoding added to the token embeddings before the attention layers. Correct
- CThe layer normalisation applied after each sublayer.
- DThe residual connections that bypass each sublayer.
Why A is wrong: Tempting because the feed-forward sublayer processes each position, but it operates identically on each position independently and does not encode order information - its absence would degrade quality but would not cause order-invariance on its own.
Why B is correct: Without positional encoding, the self-attention mechanism treats the input as a set rather than a sequence - the attention scores depend only on token identity, not position, so permuting tokens produces the same output. Positional encodings inject order information into the embeddings to break this symmetry.
Why C is wrong: Layer normalisation stabilises training and affects output magnitudes, but it contains no positional information and its removal would not cause the model to treat token order as irrelevant.
Why D is wrong: Residual connections carry gradient flow and preserve earlier representations, but they do not encode sequence order. Removing them would harm gradient propagation, not make the model order-invariant.