NCA-ADS - Introductory MLOps Practices (10% of the exam) - Section 6.4
Monitor production models for drift.
Monitor production models for data drift and concept drift by comparing incoming feature distributions and prediction quality against baselines established at training time. Determine when drift is severe enough to trigger retraining rather than a simple threshold adjustment.
A fraud-detection model was trained on transaction data from 2023. In 2025, the input feature distributions remain stable, but the relationship between the features and fraudulent behaviour has shifted because attackers have changed their tactics. Which type of drift best describes this situation, and what is the primary trigger for retraining?
AData drift, because any change in the production environment that degrades accuracy is by definition a shift in input data.
BConcept drift, because the statistical relationship between the input features and the target label has changed even though the input distributions are unchanged.check_circle Correct
CCovariate shift, because the model was trained on a sample that did not represent the full population of future transactions.
DModel degradation caused by infrastructure change, because production latency or hardware differences alter effective model outputs over time.
Distinguish concept drift from data drift by identifying which statistical distribution has changed in a given production scenario. Concept drift occurs when P(y|X) changes - the same input feature vector now corresponds to a different expected output - while the input distribution P(X) remains stable. Data drift (including covariate shift) occurs when P(X) changes. Here, attacker tactic changes alter the fraud-to-feature mapping without changing raw feature distributions, making this a clear concept drift case. Detecting concept drift typically requires monitoring model performance metrics (precision, recall, F1) against labelled ground truth, whereas data drift can be detected by comparing input feature distributions alone using tests such as the Kolmogorov-Smirnov test or Population Stability Index.
Why A is wrong: Data drift specifically refers to changes in the statistical distribution of input features, not changes in how features relate to the target label. Stable input distributions with a shifting feature-to-label relationship is a different phenomenon entirely.
Why B is correct: Concept drift describes a change in the conditional distribution P(y|X) - the mapping from features to labels - independent of any change in the marginal input distribution P(X). Attackers adopting new tactics shifts this mapping, which is exactly why the model's learned decision boundary no longer reflects reality and retraining on recent labelled data is required.
Why C is wrong: Covariate shift is a form of data drift where training and production input distributions differ. The scenario explicitly states that input feature distributions have remained stable, ruling out this category as the root cause.
Why D is wrong: Infrastructure-induced degradation relates to serving environment differences, not to a change in the underlying data-generating process. The scenario describes a behavioural shift by fraudsters, which is a data-generating-process phenomenon, not an infrastructure phenomenon.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Covariate drift, because the marginal distribution of the input features has moved away from the training distribution.
errorWhy it is wrong: Covariate (data) drift is a real category, but it requires the input distributions to shift; the team explicitly confirmed the feature histograms are unchanged, so this does not fit.
Wait for the delayed labels to accumulate, then recompute precision and recall on each fully labelled cohort before raising any alert.
errorWhy it is wrong: Labelled-cohort metrics are the gold standard for true performance, but they are exactly what the 60-to-90-day delay blocks, so they cannot provide the earliest warning the team needs.
Monitor the distribution of model output predictions over time and compare it to the training-period prediction distribution using a statistical test such as the Kolmogorov-Smirnov test.
errorWhy it is wrong: Monitoring prediction distributions can reveal distributional changes in the output space, but it conflates changes in inputs with changes in the input-to-output mapping. It does not directly confirm that the input feature distribution has shifted, which is the specific phenomenon described.