In a transformer encoder, what is the primary purpose of the multi-head attention mechanism compared to single-head attention?
- AIt allows the model to attend to information from different representation subspaces at different positions simultaneously. Correct
- BIt reduces the total number of parameters by splitting the attention matrix into independent segments.
- CIt replaces positional encoding by encoding token order directly through each attention head.
- DIt applies a recurrent connection across heads so that the output of one head is fed as input to the next head sequentially.
Why A is correct: This is the defining benefit: each head learns a distinct linear projection of queries, keys, and values, enabling the model to capture different types of relationships (syntactic, semantic, coreference) in parallel across the sequence.
Why B is wrong: Tempting because 'splitting' sounds like compression, but multi-head attention does not reduce parameters - it uses separate learned projection matrices for each head, which typically keeps or increases the parameter count relative to a single wide attention layer.
Why C is wrong: Tempting because attention heads do process position-sensitive information, but positional encoding is a separate, additive signal injected into the embeddings before attention is computed - attention heads do not replace it.
Why D is wrong: Tempting for candidates who conflate multi-head attention with sequential processing, but the heads in multi-head attention operate in parallel and independently; their outputs are concatenated, not chained recurrently.