NCA-ADS - Introductory MLOps Practices (10% of the exam) - Section 6.6
Benchmark workflows.
Benchmark data science workflows by measuring wall-clock time, GPU utilisation, and memory consumption before and after optimisation changes. Use repeatable benchmark conditions - fixed data sizes, warm-up passes, and isolated hardware - to produce comparable results.
A machine learning engineer is running a fair CPU-versus-GPU comparison for a cuML random forest training task. The first GPU run is measured at 4.2 seconds and subsequent runs average 1.1 seconds. The engineer plans to report 4.2 seconds as the representative GPU time. What is the main flaw in that plan?
AThe reported time should be the median of all runs, including the first, because the median is more robust to outliers than the mean.
BThe GPU time should be divided by the number of CPU cores to produce a per-core comparison that is fair to single-threaded CPU workloads.
CThe first run includes one-off costs such as JIT compilation and CUDA context initialisation that are not paid on subsequent calls, so using it as the representative time overstates steady-state GPU latency.check_circle Correct
DThe benchmark should be run with a larger dataset for the GPU comparison because small datasets do not saturate GPU parallelism and will always make the GPU appear slower than the CPU.
Explain why warm-up runs must be excluded from GPU benchmark results to avoid overstating steady-state latency in CPU-vs-GPU comparisons. The first execution of a GPU workload frequently triggers CUDA context creation, JIT compilation of device kernels, and driver-level initialisation. These are one-off costs that are not representative of the throughput or latency a deployed system will experience after the first call. Reporting the first-run time as the GPU's representative latency therefore overstates how slow the GPU is relative to a CPU, which does not carry these initialisation penalties in the same way. A reproducible benchmark performs at least one or two warm-up runs, discards those measurements, and reports statistics over the stable subsequent runs.
Why A is wrong: Choosing a different aggregation statistic does not address why the first run is slow; the issue is that one-off initialisation costs inflate the first measurement, and including it in any average distorts the steady-state performance the benchmark aims to characterise.
Why B is wrong: Dividing by core count is not a standard or meaningful normalisation for end-to-end training time; it conflates throughput with resource usage and does not relate to the elevated first-run time caused by warm-up overheads.
Why C is correct: GPU frameworks often compile kernels on the first invocation and initialise the CUDA context, both of which are one-time costs. Including that first run inflates the reported latency well beyond what the GPU takes for real workloads, making the comparison unfair to the GPU. Warm-up runs should be excluded from reported figures.
Why D is wrong: While dataset size does affect whether GPU parallelism is fully utilised, the question specifically asks about using the first-run time of 4.2 seconds versus the steady-state 1.1 seconds; the flaw is the one-off warm-up cost, not the dataset size.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Report the very first timed call, because it includes every cost the workload incurs and is therefore the most honest end-to-end figure for the operation.
errorWhy it is wrong: Tempting because the first call is genuinely the slowest and looks conservative, but it folds in one-time context and compilation overhead that occurs once per process rather than per operation, so it overstates the steady-state cost the user actually cares about.
GPU acceleration is constant per row, so any sample size gives the same speedup, and the only error is that 5,000 rows takes slightly longer to set up than production.
errorWhy it is wrong: Tempting because per-row reasoning sounds tidy, but GPU speedup is not constant per row: fixed overheads dominate tiny inputs while parallel throughput dominates large ones, so the ratio shifts with scale.
Replace the GPU with a more recent model so the two devices match in release year, then leave the differing input files and timing scopes unchanged.
errorWhy it is wrong: Tempting because hardware parity sounds rigorous, but matching release years does not address the core flaw of feeding the two pipelines different input data and timing different scopes, which alone make the figures incomparable.