NVIDIA-Certified Associate: Generative AI LLMs (NCA-GENL) practice questions

In a transformer encoder, what is the primary purpose of the multi-head attention mechanism compared to single-head attention?

• AIt allows the model to attend to information from different representation subspaces at different positions simultaneously.check_circle Correct
• BIt reduces the total number of parameters by splitting the attention matrix into independent segments.
• CIt replaces positional encoding by encoding token order directly through each attention head.
• DIt applies a recurrent connection across heads so that the output of one head is fed as input to the next head sequentially.

Why A is correct: This is the defining benefit: each head learns a distinct linear projection of queries, keys, and values, enabling the model to capture different types of relationships (syntactic, semantic, coreference) in parallel across the sequence.

Why B is wrong: Tempting because 'splitting' sounds like compression, but multi-head attention does not reduce parameters - it uses separate learned projection matrices for each head, which typically keeps or increases the parameter count relative to a single wide attention layer.

Why C is wrong: Tempting because attention heads do process position-sensitive information, but positional encoding is a separate, additive signal injected into the embeddings before attention is computed - attention heads do not replace it.

Why D is wrong: Tempting for candidates who conflate multi-head attention with sequential processing, but the heads in multi-head attention operate in parallel and independently; their outputs are concatenated, not chained recurrently.

A researcher notices that a transformer language model produces identical output regardless of the order in which tokens appear in the input sequence. Which component is most likely missing or misconfigured?

• AThe feed-forward sublayer within each encoder block.
• BThe positional encoding added to the token embeddings before the attention layers.check_circle Correct
• CThe layer normalisation applied after each sublayer.
• DThe residual connections that bypass each sublayer.

Why A is wrong: Tempting because the feed-forward sublayer processes each position, but it operates identically on each position independently and does not encode order information - its absence would degrade quality but would not cause order-invariance on its own.

Why B is correct: Without positional encoding, the self-attention mechanism treats the input as a set rather than a sequence - the attention scores depend only on token identity, not position, so permuting tokens produces the same output. Positional encodings inject order information into the embeddings to break this symmetry.

Why C is wrong: Layer normalisation stabilises training and affects output magnitudes, but it contains no positional information and its removal would not cause the model to treat token order as irrelevant.

Why D is wrong: Residual connections carry gradient flow and preserve earlier representations, but they do not encode sequence order. Removing them would harm gradient propagation, not make the model order-invariant.

When comparing a transformer-based language model to a recurrent neural network (RNN) for processing long text sequences, which characteristic most directly explains why transformers generally handle long-range dependencies more effectively?

• ATransformers use gating mechanisms similar to LSTM cells that selectively retain information across many time steps.
• BTransformers process sequences left-to-right one token at a time, which gives them better gradient flow than bidirectional RNNs.
• CTransformers process all tokens in direct pairwise attention in a single pass, giving any two tokens a constant path length between them.check_circle Correct
• DTransformers replace embeddings with one-hot vectors at each layer, avoiding the compression that causes RNNs to forget early tokens.

Why A is wrong: LSTM-style gating is an RNN mechanism, not a transformer one. Transformers do not use gates; they use attention weights to route information. Attributing LSTM gating to transformers conflates two separate architectures.

Why B is wrong: Transformers do not process tokens sequentially left-to-right; they process the entire sequence in parallel. Claiming they are sequential conflates transformer inference with autoregressive generation - and bidirectional RNNs already improve over unidirectional ones for gradient flow.

Why C is correct: The scaled dot-product attention computes relationships between every pair of positions simultaneously, so the path length between any two tokens is O(1) regardless of sequence length. In an RNN, the signal from an early token must pass through every intermediate hidden state, creating O(n) path length and vanishing-gradient risk.

Why D is wrong: Transformers use dense learned embeddings, not one-hot vectors, at their input layer. One-hot representations are extremely high-dimensional and sparse; replacing embeddings with them would not improve memory of long-range dependencies.