ExamworthyExamworthy
Open the appStart free

NCA-ADS - Introductory MLOps Practices - Section 6.6

Benchmark workflows.

Benchmark data science workflows by measuring wall-clock time, GPU utilisation, and memory consumption before and after optimisation changes. Use repeatable benchmark conditions - fixed data sizes, warm-up passes, and isolated hardware - to produce comparable results.

Practice question for this objective

Free sampleIntroductory MLOps Practicesmedium

A machine learning engineer is running a fair CPU-versus-GPU comparison for a cuML random forest training task. The first GPU run is measured at 4.2 seconds and subsequent runs average 1.1 seconds. The engineer plans to report 4.2 seconds as the representative GPU time. What is the main flaw in that plan?

  • AThe reported time should be the median of all runs, including the first, because the median is more robust to outliers than the mean.
  • BThe GPU time should be divided by the number of CPU cores to produce a per-core comparison that is fair to single-threaded CPU workloads.
  • CThe first run includes one-off costs such as JIT compilation and CUDA context initialisation that are not paid on subsequent calls, so using it as the representative time overstates steady-state GPU latency. Correct
  • DThe benchmark should be run with a larger dataset for the GPU comparison because small datasets do not saturate GPU parallelism and will always make the GPU appear slower than the CPU.
Explain why warm-up runs must be excluded from GPU benchmark results to avoid overstating steady-state latency in CPU-vs-GPU comparisons. The first execution of a GPU workload frequently triggers CUDA context creation, JIT compilation of device kernels, and driver-level initialisation. These are one-off costs that are not representative of the throughput or latency a deployed system will experience after the first call. Reporting the first-run time as the GPU's representative latency therefore overstates how slow the GPU is relative to a CPU, which does not carry these initialisation penalties in the same way. A reproducible benchmark performs at least one or two warm-up runs, discards those measurements, and reports statistics over the stable subsequent runs.

Why A is wrong: Choosing a different aggregation statistic does not address why the first run is slow; the issue is that one-off initialisation costs inflate the first measurement, and including it in any average distorts the steady-state performance the benchmark aims to characterise.

Why B is wrong: Dividing by core count is not a standard or meaningful normalisation for end-to-end training time; it conflates throughput with resource usage and does not relate to the elevated first-run time caused by warm-up overheads.

Why C is correct: GPU frameworks often compile kernels on the first invocation and initialise the CUDA context, both of which are one-time costs. Including that first run inflates the reported latency well beyond what the GPU takes for real workloads, making the comparison unfair to the GPU. Warm-up runs should be excluded from reported figures.

Why D is wrong: While dataset size does affect whether GPU parallelism is fully utilised, the question specifically asks about using the first-run time of 4.2 seconds versus the steady-state 1.1 seconds; the flaw is the one-off warm-up cost, not the dataset size.

See more NCA-ADS practice questions, answers explained.

More in this domain

Back to all Introductory MLOps Practices objectives, or the NCA-ADS cert hub.

Examworthy is not affiliated with or endorsed by NVIDIA. Original, blueprint-aligned practice material only.