PMLE - Scaling Prototypes Into ML Models (21% of the exam) - Section 3.3

Choose appropriate hardware for training, evaluating CPU, GPU, and TPU options and understanding distributed training across GPUs and TPUs using data and model parallelism strategies.

Compare CPU, GPU, and TPU options for ML training workloads, recognising the throughput and memory trade-offs that make each accelerator appropriate for a given model size and batch size. Distinguish data parallelism from model parallelism and understand how each strategy distributes computation across multiple devices during distributed training.

CPU, GPU, and TPUDistributed trainingData and model parallelism

Practice question for this objective

Free sampleScaling Prototypes Into ML Modelsmedium

An engineer is documenting how data parallelism and model parallelism differ before configuring a distributed Vertex AI custom training job. Which TWO statements correctly characterise these distributed training strategies? (Select TWO.)

AData parallelism is the appropriate choice when the model's weights and optimiser state are too large to fit in a single accelerator's memory even at batch size one.
BIn data parallelism each worker holds a complete replica of the model and processes a different shard of the input batch, synchronising gradients between the workers. Correct
CModel parallelism speeds up training mainly by giving every device a full copy of the network and feeding each device a separate slice of the dataset.
DData parallelism reduces each device's memory footprint by storing only a fraction of the model's parameters and optimiser state on every individual worker.
EModel parallelism partitions the model itself, placing different layers or weight tensors on different devices so a model too large for one device can train at all. Correct

Data parallelism replicates the whole model and shards the data, whereas model parallelism splits the model across devices so oversized models can train. Data parallelism gives every worker a full model replica and a different data shard, then synchronises gradients, so it scales throughput but not capacity. Model parallelism splits the model's layers or weight tensors across devices, which is the only way to train a model that cannot fit on a single accelerator.

Why A is wrong: It is tempting because both strategies scale training, but replicating a model that already does not fit cannot help; that memory limit calls for model parallelism instead.

Why B is correct: This is the defining property of data parallelism: full model copies per worker, different data shards, with gradient synchronisation such as all-reduce.

Why C is wrong: This sounds reasonable but actually describes data parallelism; model parallelism splits one model across devices rather than replicating it.

Why D is wrong: It tempts as a memory benefit, but data parallelism replicates the whole model per worker, so it does not cut per-device parameter memory; that is a model-parallel property.

Why E is correct: Splitting the model's parameters across devices is exactly what model parallelism does, enabling training of models that exceed a single device's memory.

See more PMLE practice questions, answers explained.

Exam traps in Scaling Prototypes Into ML Models

Answers that look right on this material and are not. Each one is a distractor from a different question in the PMLE bank for this domain.

Pure data parallelism, replicating the full model on every GPU and giving each a different data shard.
Why it is wrong: This is tempting because data parallelism shortens training over large datasets, but it requires each GPU to hold the whole model, which is impossible here.
The Cloud Storage bucket holding the training shards is in a different region from the training cluster, so cross-region reads are exhausting GPU memory before the optimiser runs.
Why it is wrong: Cross-region bucket placement is a genuine performance and cost mistake that prepared candidates worry about, but it adds network latency and egress cost rather than consuming GPU memory, so it cannot explain a CUDA out-of-memory failure.
Model parallelism, splitting the model layers across the eight GPUs so each device holds only a fraction of the parameters during every step.
Why it is wrong: It is tempting because spreading work across GPUs sounds like the goal, but model parallelism targets models too large for one device's memory and adds inter-device communication overhead, which is unnecessary here since the model already fits on one GPU.

Examworthy is not affiliated with or endorsed by Google Cloud. Original, blueprint-aligned practice material only.