During a large distributed training job, one GPU node begins reporting elevated GPU memory errors and reduced throughput compared to its peers. From a cluster health-monitoring perspective, which action best addresses early detection of this type of degradation before it causes a job failure?
- AIncrease the batch size on the affected node to verify whether performance recovers under a heavier load.
- BConfigure threshold-based alerts on GPU memory error counters and utilisation metrics so the operations team is notified while the node can still be investigated gracefully. Correct
- CRely on the training framework to raise an out-of-memory exception when the fault becomes critical.
- DRestart the GPU driver on the affected node immediately without capturing any telemetry.
Why A is wrong: Increasing batch size adds memory pressure and compute load to a node already showing degraded behaviour, which is likely to accelerate failure rather than diagnose or resolve it.
Why B is correct: Threshold-based alerting on telemetry counters such as GPU memory errors enables proactive intervention - draining the node, running diagnostics, and replacing hardware - before a job failure propagates across the cluster.
Why C is wrong: Waiting for a framework-level exception is reactive, not proactive; by that point the job has already failed and cluster time has been wasted, which is precisely what health monitoring aims to prevent.
Why D is wrong: Restarting the driver without first capturing telemetry discards diagnostic information and does not confirm whether the issue is transient or indicative of hardware failure requiring node replacement.