Free NCA-GENL flashcards

In a transformer encoder, what is the primary purpose of multi-head attention compared to single-head attention?

Multi-head attention lets the model attend to information from different representation subspaces at different positions simultaneously. Each head learns a distinct linear projection of queries, keys, and values, so a single layer can capture different relationship types such as syntactic dependencies and semantic relatedness in parallel.

Multi-head attention reduces the model's parameter count by splitting the attention matrix into independent segments.

Splitting sounds like compression, but multi-head attention does not reduce parameters. It uses separate learned projection matrices for each head, which typically keeps or increases the parameter count relative to a single wide attention layer.

In channels-last (NHWC) format, what tensor shape represents a batch of 32 greyscale 256x256 images fed to the first conv layer?

(32, 256, 256, 1). NHWC orders axes as (batch, height, width, channels). A greyscale image has exactly one channel, so the channel axis is placed last with value 1, matching the expected input shape in channels-last mode.

Replacing feed-forward layers with mixture-of-experts (MoE) blocks fixes the quadratic self-attention bottleneck.

MoE cuts feed-forward cost by routing tokens to specialist sub-networks, but the self-attention bottleneck is a separate mechanism. MoE does not change how attention scores are computed across the sequence length, so it leaves the O(n^2) scaling intact.

In a PyTorch DataLoader, how do num_workers and pin_memory reduce inter-batch GPU idle time?

With num_workers=0 the main thread loads each batch synchronously, leaving the GPU idle during preparation. Setting num_workers greater than zero spawns worker processes that prefetch the next batch while the GPU trains, and pin_memory=True places output tensors in page-locked host memory so the CUDA DMA engine transfers them asynchronously at higher bandwidth. Together they overlap CPU loading with GPU computation.

Increasing batch size substantially fixes GPU idle time between batches by amortising transfer overhead.

A larger batch size reduces the number of host-to-device transfers per epoch but introduces no prefetching. The root cause is that data loading is sequential and blocking, which batch size does not change. It may also cause out-of-memory errors or destabilise training.

Which text augmentation technique increases training examples while preserving the original label for a text classifier with few labelled samples?

Synonym replacement, which swaps selected words with meaning-equivalent alternatives from a lexical database. Because the substitution is meaning-preserving, the class label stays valid while the model sees a distinct token sequence it has not encountered, addressing data scarcity without new annotations.

Randomly shuffling sentence order within each document is a safe, label-preserving augmentation for text classification.

Shuffling sentences creates new orderings but disrupts coherence and often changes meaning enough to corrupt the original label, making it unreliable for classification. A meaning-preserving method like synonym replacement keeps the label intact instead.