A research team is training a model whose parameter count is so large that the weights and optimiser state cannot fit into the memory of any single accelerator they have available, even at a batch size of one. They have a multi-device TPU slice and need a strategy that lets the model train at all. Which approach is appropriate?
- AData parallelism, replicating the full model on every device and sharding the input batch so each device trains on a different subset of examples.
- BModel parallelism, partitioning the model's layers and parameters across the devices so each holds only part of the network and they cooperate on each step. Correct
- CGradient accumulation on one device, summing gradients over several micro-batches before applying a single optimiser update to the weights.
- DMixed-precision training on one device, storing weights in lower precision so the existing single accelerator can hold the whole model.
Why A is wrong: It is tempting because data parallelism is the common default for scaling, but it requires a full copy of the model on every device, which is impossible here because the model does not fit on a single accelerator.
Why B is correct: Model parallelism splits the parameters and layers across devices so no single device must hold the whole model, which is the only way to train when the network exceeds one accelerator's memory even at batch size one.
Why C is wrong: It is tempting because accumulation reduces memory pressure from activations, but it still keeps the entire model and optimiser state on one device, so it cannot help when the parameters themselves do not fit.
Why D is wrong: It is tempting because lower precision does shrink memory use, but the reduction is modest and will not make a model that overflows the device at batch size one suddenly fit, so a single device remains insufficient.