An engineer is documenting how data parallelism and model parallelism differ before configuring a distributed Vertex AI custom training job. Which TWO statements correctly characterise these distributed training strategies? (Select TWO.)
- AData parallelism is the appropriate choice when the model's weights and optimiser state are too large to fit in a single accelerator's memory even at batch size one.
- BIn data parallelism each worker holds a complete replica of the model and processes a different shard of the input batch, synchronising gradients between the workers. Correct
- CModel parallelism speeds up training mainly by giving every device a full copy of the network and feeding each device a separate slice of the dataset.
- DData parallelism reduces each device's memory footprint by storing only a fraction of the model's parameters and optimiser state on every individual worker.
- EModel parallelism partitions the model itself, placing different layers or weight tensors on different devices so a model too large for one device can train at all. Correct
Why A is wrong: It is tempting because both strategies scale training, but replicating a model that already does not fit cannot help; that memory limit calls for model parallelism instead.
Why B is correct: This is the defining property of data parallelism: full model copies per worker, different data shards, with gradient synchronisation such as all-reduce.
Why C is wrong: This sounds reasonable but actually describes data parallelism; model parallelism splits one model across devices rather than replicating it.
Why D is wrong: It tempts as a memory benefit, but data parallelism replicates the whole model per worker, so it does not cut per-device parameter memory; that is a model-parallel property.
Why E is correct: Splitting the model's parameters across devices is exactly what model parallelism does, enabling training of models that exceed a single device's memory.