PMLE - Monitoring AI Solutions (13% of the exam) - Section 6.2

Monitor, test, and troubleshoot AI solutions using Model Monitoring on the Agent Platform for continuous evaluation, detecting common issues such as training-serving skew, data drift, concept drift, and feature attribution drift, and evaluating gen AI solutions.

Use Model Monitoring on the Agent Platform to continuously evaluate deployed models and detect training-serving skew, data drift, concept drift, and feature attribution drift before they degrade production quality. Apply structured evaluation methods for generative AI solutions, interpreting monitoring signals to decide when retraining or rollback is warranted.

Model MonitoringTraining-serving skewData and concept driftGen AI evaluation

Practice question for this objective

Free sampleMonitoring AI Solutionsmedium

A retail forecasting model has run in production on Vertex AI for three months. The serving features come from a streaming pipeline, while the model was trained on a batch snapshot of the same source. Latency-sensitive predictions have grown steadily less accurate, but the live input feature distributions still match the training data when checked against the original training baseline. Which Model Monitoring signal is the team most likely observing, and what does it indicate?

AFeature attribution drift, indicating that the relative importance the model places on each feature has shifted even though raw inputs look stable.
BTraining-serving skew, indicating that the feature values produced at serving time differ from those produced by the training pipeline.
CConcept drift, indicating that the statistical relationship between the input features and the target has changed over time. Correct
DData drift, indicating that the distribution of the incoming serving features has moved away from the training distribution.

Distinguish concept drift from data drift and skew by recognising that concept drift degrades accuracy while input distributions stay stable. Concept drift is a change in the conditional relationship between features and the target, so the same inputs map to different outcomes over time; accuracy falls even though input distributions and the training-serving pipeline remain aligned.

Why A is wrong: This is tempting because attribution drift can occur while inputs look stable, but attribution drift reflects changes in how features contribute to predictions, not a change in the input-to-target relationship that degrades accuracy with unchanged inputs and unchanged outputs.

Why B is wrong: Skew is plausible given the batch-versus-streaming pipeline split, but the stem states serving feature distributions still match the training baseline, so the inputs are not diverging between training and serving.

Why C is correct: Inputs match the training baseline yet accuracy falls, so the mapping from features to target has changed, which is the definition of concept drift and explains degradation without input distribution change.

Why D is wrong: Data drift is a natural first guess for falling accuracy, but it requires the serving input distribution to shift, and the stem explicitly says inputs still match the training baseline, ruling it out.

See more PMLE practice questions, answers explained.

Exam traps in Monitoring AI Solutions

Answers that look right on this material and are not. Each one is a distractor from a different question in the PMLE bank for this domain.

The raw input data has shifted so far that the model is now receiving values it never saw during training, even though the thresholds have not yet fired.
Why it is wrong: Tempting because attribution changes often follow input shifts, but the premise states each feature distribution is within threshold, so the inputs themselves have not materially shifted.
Generative models do not produce numeric features, so Model Monitoring cannot compute any distribution over their inputs and must rely solely on manual human review.
Why it is wrong: Tempting because inputs are text, but prompts can still be characterised statistically; the real issue is scoring the output quality, and automated evaluators exist beyond manual review.
It signals that the serving pipeline is transforming features inconsistently with the training pipeline for every request.
Why it is wrong: Tempting because both relate to feature behaviour, but that description is training-serving skew, a pipeline mismatch, rather than a change in how features contribute to outputs.

Examworthy is not affiliated with or endorsed by Google Cloud. Original, blueprint-aligned practice material only.