Core Machine Learning and AI Knowledge

RoBERTa modifies BERT's pre-training procedure to improve downstream performance. Which single change in RoBERTa most directly addresses a known limitation in BERT's original self-supervised objective design?

• AReplacing the WordPiece tokeniser with a byte-pair encoding tokeniser trained on a larger vocabulary, reducing unknown-token frequency.
• BRemoving the next-sentence prediction (NSP) objective entirely, because empirical results showed NSP did not improve - and sometimes harmed - downstream task performance.check_circle Correct
• CIntroducing a span-level masking strategy in which contiguous spans of tokens are masked together, replacing token-level random masking.
• DAdding a sentence-order prediction (SOP) objective to replace NSP, training the model to distinguish correct sentence order from swapped sentence order.

Why A is wrong: RoBERTa does adopt byte-pair encoding, but this is a tokenisation engineering choice. It does not address a fundamental limitation of BERT's self-supervised objectives (MLM and NSP), which is the core concern of the question.

Why B is correct: NSP was introduced in BERT to help sentence-pair tasks, but RoBERTa's authors found that models trained without NSP matched or exceeded BERT on benchmarks. NSP mixes documents and encourages the model to exploit topic differences rather than genuine inter-sentence reasoning, making it a flawed objective. Removing it is the primary objective-level improvement.

Why C is wrong: Span masking is the strategy introduced in SpanBERT, not RoBERTa. Attributing it to RoBERTa is a plausible confusion given both are BERT variants, but RoBERTa's key objective change is dropping NSP, not altering the masking granularity.

Why D is wrong: Sentence-order prediction is the replacement strategy used in ALBERT, not RoBERTa. ALBERT found that SOP is a harder and more meaningful objective than NSP. RoBERTa's approach was to remove NSP altogether rather than replace it with a harder variant.

A team scales a single transformer layer across 8 GPUs using Megatron-style tensor parallelism, sharding the weight matrices of the feed-forward and attention blocks. During profiling they find that the all-reduce communication between shards dominates wall-clock time once they cross a single server boundary. Given how tensor parallelism communicates, what is the most defensible explanation for the steep cost after crossing that boundary?

• ATensor parallelism synchronises only once per training step at the optimiser update, so crossing nodes adds a single extra collective that scales with model size.
• BTensor parallelism requires an all-reduce on activations within every forward and backward pass of each layer, so placing shards across slower inter-node links multiplies that per-layer synchronisation over the lower-bandwidth interconnect.check_circle Correct
• CTensor parallelism transfers full layer weights between GPUs on every microbatch, so the cost rises because weight tensors are larger than activation tensors.
• DTensor parallelism uses point-to-point sends between consecutive pipeline stages, so spanning nodes increases the bubble created by stage-to-stage handoffs.

Why A is wrong: Tempting because once-per-step synchronisation describes data parallelism, but tensor parallelism communicates multiple times per layer per pass, which is why it is far more sensitive to interconnect speed.

Why B is correct: Correct: tensor parallelism injects collective communication inside each layer, so the high frequency of all-reduces makes it sensitive to the bandwidth of the link the shards span, which drops sharply between nodes.

Why C is wrong: Tempting because moving weights sounds expensive, but tensor parallelism keeps each weight shard resident on its GPU and communicates activations, not full weight matrices, between shards.

Why D is wrong: Tempting because pipeline bubbles are a real cost, but point-to-point stage handoffs and bubbles describe pipeline parallelism; tensor parallelism relies on all-reduce collectives within a layer.

A practitioner is training a large language model from scratch and notices that gradients in the earliest layers are effectively zero after several epochs. Which technique most directly addresses this problem?

• AApply gradient clipping with a threshold of 1.0 to prevent excessively large gradient values.
• BReduce the number of hidden layers so the backpropagation path is shorter.
• CIncrease the weight initialisation scale so initial activations are larger at training start.
• DReplace saturating activation functions with ReLU and introduce residual skip connections.check_circle Correct

Why A is wrong: Tempting because gradient clipping is a common training stability tool, but it is designed to prevent exploding gradients, not vanishing ones. Clipping a near-zero gradient has no effect on the underlying vanishing gradient problem caused by repeated multiplication of small values through many layers.

Why B is wrong: Tempting because fewer layers means fewer multiplications during backpropagation, which would reduce gradient vanishing. However, reducing depth sacrifices model capacity, and the standard industry approach is to preserve depth while using architectural features that maintain gradient magnitude, not to shrink the model.

Why C is wrong: Tempting because poor weight initialisation can exacerbate vanishing gradients early in training. However, simply increasing the initialisation scale without a principled scheme such as He or Xavier initialisation risks pushing activations into saturation, and does not solve the structural problem once training has progressed past the early epochs where the symptom is now observed.

Why D is correct: Saturating activations such as sigmoid compress gradients toward zero in their saturation regions. ReLU has a gradient of exactly 1 in the positive region, preventing compression. Residual connections provide a direct gradient path to earlier layers, bypassing the multiplicative chain, which is the primary architectural solution to vanishing gradients in deep networks.

A network engineer analyses a communications graph to find the routers whose removal would most disrupt traffic, because a large share of shortest paths between other router pairs passes through them. Which centrality measure most directly identifies these critical bridging routers?

• ADegree centrality, which scores a node by the raw count of edges directly attached to it.
• BCloseness centrality, which scores a node by the inverse of its average shortest-path distance to all others.
• CBetweenness centrality, which scores a node by the fraction of all-pairs shortest paths that run through it.check_circle Correct
• DEigenvector centrality, which scores a node higher when it is connected to other high-scoring nodes.

Why A is wrong: Degree centrality is tempting because well-connected routers often matter, but a router can have many local links yet sit off the main shortest paths, so a high degree does not guarantee that removing it disrupts cross-network traffic.

Why B is wrong: Closeness centrality rewards nodes that can reach everything quickly and uses shortest paths, which makes it look applicable, but it measures a node's own proximity rather than how many paths between other nodes depend on it.

Why C is correct: Betweenness centrality counts, for every pair of other nodes, the proportion of shortest paths passing through a given node, so it precisely measures how much traffic must route through a node and flags the bridging routers whose loss is most disruptive.

Why D is wrong: Eigenvector centrality captures influence through well-connected neighbours and resembles PageRank, so it is plausible, but it reflects prestige among neighbours rather than the bridging role of carrying shortest-path traffic between separate parts of the network.