NCA-ADS - Introductory MLOps Practices (10% of the exam) - Section 6.1
Monitor data science pipelines.
Monitor data science pipelines by tracking resource utilisation, execution times, and error rates at each stage to identify bottlenecks and failures. Distinguish between pipeline-level monitoring, which covers workflow health, and model-level monitoring, which covers prediction quality.
A deployed regression model predicts daily energy demand, and the actual demand for any given day is only known once metered readings settle two days later. A monitoring engineer wants a single dashboard panel that tracks how the model's prediction error evolves day by day, accounting correctly for the fact that today's error cannot be computed until its ground truth lands. Which approach correctly tracks live prediction error under this delay?
APlot the model's error each day against the predictions made that same day before any ground truth has arrived.
BTrack only the distribution of the model's raw output values and treat any change in their spread as the error metric.
CCompute error only at the end of each month once all labels are final, and leave the panel blank until then.
DJoin each prediction to its ground truth once the metered reading settles, then plot the error on a rolling window aligned to the prediction date.check_circle Correct
Track live prediction error under delayed labels by joining each prediction to its ground truth as it matures and plotting a rolling error aligned to the prediction date. With delayed ground truth, error can only be computed once labels settle. Backfilling each prediction with its matured label and aligning a rolling error to the prediction date yields a faithful, timely performance trend rather than a fabricated same-day figure or a once-a-month lump.
Why A is wrong: Charting error on the day of prediction is tempting for immediacy, but the true demand is not yet known, so any error computed then is undefined or fabricated and cannot reflect real performance.
Why B is wrong: Watching the spread of outputs is a reasonable drift check, but the spread of predictions is not the prediction error; the model could be confidently wrong with a perfectly stable output distribution.
Why C is wrong: Waiting for a full month of settled labels gives accurate numbers but no timely signal, so a sustained mid-month degradation would stay invisible far longer than the two-day delay actually requires.
Why D is correct: Joining each prediction to its later-arriving ground truth and plotting a rolling error aligned to the prediction date correctly handles the two-day delay, giving an honest, continuously updating view of real prediction error as labels mature.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Track model prediction-score distributions using a statistical drift detector applied after each batch of inference results.
errorWhy it is wrong: Prediction-score drift detectors operate on model outputs, not on intermediate pipeline stages, so a silent zero-row join upstream would not be visible until predictions are already corrupted, making this a model-health tool rather than a pipeline-health tool.
Emit metrics and failures only to the pipeline's standard output log, and have the on-call engineer read the log file each morning.
errorWhy it is wrong: A single log read in the morning is simple, but it provides no immediate notification, so an overnight failure would go unaddressed for hours, which defeats the requirement for a prompt page on failure.
Measure the ratio of GPU worker idle time to active compute time across the Dask cluster during each pipeline run.
errorWhy it is wrong: Worker idle-to-active ratio is a resource-efficiency indicator that helps optimise cost and throughput, but it does not capture whether the pipeline completed and delivered results within the required freshness window, so stale dashboards could persist even when GPU utilisation looks healthy.