NCA-AIIO - AI Operations (22% of the exam) - Section 3.2
Describe AI cluster orchestration and job scheduling essentials.
Describe how orchestration platforms such as Kubernetes with the NVIDIA GPU Operator manage containerised AI workloads, and how job schedulers such as Slurm allocate GPU resources across multi-tenant clusters. Distinguish between queue-based scheduling for batch training jobs and priority-based scheduling needed to meet latency requirements for interactive workloads.
OrchestrationJob scheduling
Practice question for this objective
lock_openFree sampleAI Operationsmedium
A platform team is migrating containerised inference services to a Kubernetes cluster that has GPUs. Multiple small inference replicas each need a fraction of a single GPU rather than an entire device. Which approach best enables multiple pods to share one physical GPU on that cluster?
ARequest multiple CPU cores per pod and rely on the GPU driver to time-share the device automatically across pods
BSet Kubernetes CPU and memory requests per pod and let the default scheduler bin-pack the inference replicas onto the GPU node with no GPU-sharing driver
CDeploy a fractional GPU scheduler such as Run:ai or NVIDIA MIG, which partitions the GPU and exposes sub-GPU resource fractions to the Kubernetes schedulercheck_circle Correct
DSet resource requests to zero for the GPU field so pods schedule freely, then configure a node taint to attract only inference workloads
Describe how fractional GPU scheduling enables multiple Kubernetes pods to share a single physical GPU for inference workloads. Standard Kubernetes GPU device plugins expose whole GPUs as integer resources, so each pod receives one or more complete devices. Fractional GPU solutions such as the Run:ai scheduler or NVIDIA MIG break this constraint. Run:ai provides a virtual GPU abstraction that enforces per-pod memory and compute limits on a shared physical GPU. MIG statically partitions supported NVIDIA GPUs into isolated instances with their own memory bandwidth and compute resources, each presented to Kubernetes as a separate allocatable resource. Both approaches allow multiple smaller inference pods to coexist on one GPU, improving utilisation without interference.
Why A is wrong: Standard GPU drivers do not automatically time-share a GPU across independent processes in proportion to pod CPU requests. Without an explicit partitioning or virtualisation layer, each pod would attempt to use the full device, causing contention.
Why B is wrong: CPU and memory requests constrain only host resources; they do not partition or arbitrate the GPU. With no GPU-sharing driver the default scheduler still treats the GPU as one indivisible device, so the first pod claims it and the remaining replicas are starved or fail to schedule.
Why C is correct: Fractional GPU schedulers extend the Kubernetes device-plugin model to advertise sub-GPU resources. Run:ai intercepts scheduling and enforces GPU memory and compute quotas per pod. NVIDIA Multi-Instance GPU (MIG) partitions a physical GPU into independent instances with dedicated memory and compute slices, each schedulable as a distinct Kubernetes resource.
Why D is wrong: Setting GPU requests to zero causes the Kubernetes scheduler to ignore GPU availability entirely; pods would be placed without any GPU allocation or enforcement, leading to uncontrolled device access and potential out-of-memory failures.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-AIIO bank for this domain.
Set both teams to the same fixed wall-clock run-time limit so jobs rotate predictably through the queue
errorWhy it is wrong: A shared run-time limit is tempting for fairness, but it does not give interactive jobs faster access and would force long training jobs to checkpoint and requeue unnecessarily.
Horizontal pod autoscaling, which adds worker replicas automatically when the existing pods report high utilisation
errorWhy it is wrong: Autoscaling is tempting because it also manages replica counts, but it reacts to load metrics on already-running pods and does nothing to guarantee that a fixed worker set starts together, so the partial-placement deadlock remains.
Add a resource request for GPUs to every CPU pod so the scheduler accounts for them on the GPU nodes
errorWhy it is wrong: Forcing CPU pods to request GPUs is tempting because GPU resource accounting drives placement, but it would make those pods consume GPU resources they do not need and could even pull them onto GPU nodes, worsening rather than solving the contention.