Model Monitoring in Production: Drift, Decay and Alerting

September 10, 2026

Model monitoring in production exists to answer one question: is the model still doing the job it was deployed to do? Drift answers a different one, whether the world the model sees has changed, so an alert on drift is not an alert on a problem. Teams start with drift because it can be computed on day one, then get either alerts that mean nothing or silence while a model ages on stable inputs.

Vela and colleagues (Scientific Reports, 2022) ran four standard models on 32 datasets from four industries and observed temporal quality degradation in 91% of the 128 model-dataset pairs, a result about those pairs rather than all production models. Google Cloud's MLOps guidance notes that models "can decay in more ways than conventional software systems" and need monitoring to "send notifications or roll back when values deviate from your expectations". Alert on the symptom, open a ticket on the cause.

What Model Monitoring in Production Has to Catch

Model monitoring in production is the continuous comparison of a model's inputs, predictions and, where available, outcomes against a baseline, so that a drop in quality becomes a decision before it becomes a business incident. Application monitoring tells you the endpoint answered. Model monitoring tells you whether the answer is still good. Latency, errors and uptime already sit in your infrastructure monitoring.

Azure Machine Learning's documentation lists shifts in data distribution, training-serving skew, data quality problems, and changes in the environment and in consumer behavior as reasons a model goes "stale".

This article is about deciding what to watch once a model is live and what should wake someone up: which signals you can measure given how late your labels arrive, which baseline each signal needs, and how to turn a threshold breach into a retrain, rollback or retire decision. It does not re-explain data drift versus concept drift, it does not cover data contracts or pipeline testing, and it does not cover evaluating LLMs, where the label never arrives at all.

Drift Is a Cause, Decay Is a Symptom

A shift in the data is not proof of harm. In Failing Loudly (NeurIPS 2019), Rabanser, Günnemann and Lipton note that ML systems "tend to fail silently" and treat detecting a shift and judging whether it is malignant as separate problems. Domain-discriminating approaches, they report, help with the second.

The reverse holds too. Vela and colleagues set out to study how degradation "can develop even under minimal drifts in the data", and they write that there is still no systematic answer on what should trigger retraining "and whether it is needed at all".

The Google SRE book already frames monitoring around two questions, "what's broken, and why?", the symptom first and the cause second. For models, decay is the symptom and justifies an alert. Drift is a candidate cause and justifies an investigation. Page on every drift score and engineers learn to ignore pages; watch drift alone and you miss the model that ages on unchanged inputs.

Label Latency Decides What You Can Measure

What you can measure depends on when ground truth arrives: within minutes, weeks later, or never.

With fast labels, you measure realized performance directly. Azure recommends the model performance signal whenever ground truth is available, since it "provides an objective view of model performance in production".

With labels delayed by weeks or months, as in credit, churn or fraud, estimated performance is the bridge. NannyML's documentation describes estimating performance "in the absence of ground truth", with CBPE for classification and DLE for regression. CBPE assumes well-calibrated probabilities, and "CBPE will not work under concept drift", which usually cannot be identified without labels. Estimation shortens the blind period; it does not replace labels.

When labels never arrive, only proxies remain: input, prediction and feature attribution drift. Mougan's 2025 doctoral thesis calls performance metrics in this regime hard to compute, "even often impossible". For generative systems, evaluation replaces monitoring as the hard problem, which is the subject of evaluating LLMs without ground truth.

So ask first: how many days pass between a prediction and the moment you know whether it was right? That number shapes model monitoring in production more than any vendor choice.

Four Signals and the Baseline Each One Needs

A signal without a baseline is a line on a chart. Four signals cover most predictive models, plus an optional fifth. The baseline column follows Azure and Google Cloud; the route column is our recommendation.

Signal	What it answers	Baseline	Needs labels?	Default route
Data quality	Is the input broken right now?	Training data	No	Page if it corrupts live predictions, else ticket
Input drift and skew	Has what the model sees moved?	Training data for skew, a past production window for drift	No	Ticket
Prediction drift	Has what the model outputs moved?	Validation data	No	Ticket
Estimated performance	Is quality probably falling?	Validation performance	No, needs calibrated scores	Page if sustained below threshold
Realized performance	Is quality falling?	Validation performance or an agreed threshold	Yes	Page if sustained below threshold
Feature attribution drift (optional)	Has what drives predictions changed?	Feature importance at training time	No	Dashboard

Google's Vertex AI Model Monitoring, now part of its Agent Platform, separates training-serving skew (production features departing from the training distribution) from drift (the production distribution changing over time). Azure baselines data drift and data quality on training data and prediction drift on validation data, and pairs data drift with feature attribution drift for "early warnings about model performance problems".

The list belongs to no single vendor. Azure names five signals. Amazon SageMaker Model Monitor names data quality, model quality, bias drift and feature attribution drift, though AWS says the service "is no longer open to new customers", so it is only an example here. Google covers input feature drift, output inference drift and feature attribution. Databricks data profiling profiles an inference table of inputs, predictions and optional labels against an optional baseline table. Four platforms, one short list.

Thresholds That Do Not Page You for Nothing

Most drift noise comes from defaults nobody chose. In Evidently's documentation, up to 1,000 reference observations means Kolmogorov-Smirnov for numerical columns and chi-squared for categorical ones, with drift at p ≤ 0.05. Above 1,000, it switches to Wasserstein distance and Jensen-Shannon divergence with a 0.1 threshold. Our reading, which Evidently does not state: on large samples a significance test flags differences too small to matter, while a distance tracks the size of the change.

Azure advises per-metric thresholds set by people who know the model, "to avoid alert fatigue". It also suggests monitoring the top N features or a subset to cut computation cost and noise, and matching check frequency to data volume.

Beyond the docs, small windows raise false alarms, and a population average can hide a segment that has already broken. Use windows large enough to be stable and check business-critical segments separately. A threshold nobody can justify is a default, not a decision.

Alerts People Act On: Page, Ticket or Dashboard

Most guides stop at "set up alerts". Site reliability engineering went further. The SRE book asks whether a rule catches a condition that is "urgent, actionable, and actively or imminently user-visible", and says "every page should be actionable"; a page that merits only a robotic response should not be a page. The SRE Workbook scores an alerting rule on precision, recall, detection time and reset time. Treating a model quality target as an SLO, with burn-rate alerting, is our adaptation.

Applied to model monitoring in production, that gives three routes:

Page: realized or estimated performance below the agreed threshold across the window, or a broken input corrupting predictions now.
Ticket: drift in inputs or predictions with no sign of degradation, handled in working hours.
Dashboard: trends, segments and attributions.

Every model needs a named owner and an escalation path. The NIST AI Risk Management Framework, voluntary and non-regulatory, names the duty: under MEASURE 2.4, AI system functionality and behavior "are monitored when in production", and MANAGE 4.1 calls for post-deployment monitoring plans covering "incident response, recovery, and change management".

From Alert to Decision: Retrain, Roll Back or Retire

An alert that ends in a chat thread has not ended. Each page should close with a decision.

Retrain. Google Cloud's MLOps guidance lists the triggers: on demand, on a schedule, on new training data, on performance degradation, and on significant changes in data distributions. Degradation is evidence and drift is a hypothesis, so a model retrained on drift alone should pass validation before it replaces anything.

Roll back. The same guidance keeps a pointer to the previous trained model for rollback, which works only if that version is ready before the new one ships.

Retire. NIST MANAGE 2.4 asks for mechanisms to "supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use". A model that retraining cannot fix should be switched off deliberately instead of left to rot.

Five questions show whether your model monitoring in production will hold:

How many days pass between a prediction and its label?
Which signal wakes a person, and which only opens a ticket?
What is the baseline for each signal, and who approved it?
Who owns the model at 3 a.m.?
Is the previous version ready to restore?

This is the work we do with clients. For the monitoring layer itself, see our ML model monitoring services; if monitoring is one gap in a wider delivery process, it belongs in an end-to-end MLOps practice. If your models are live and nobody can answer the first question, talk to our MLOps engineers.

Frequently Asked Questions

What is model monitoring in production?

Model monitoring in production is the continuous comparison of a model's inputs, predictions and outcomes against a baseline, to catch quality loss and turn it into a decision. Unlike application monitoring, it checks whether the answer is good, not only whether an answer arrived.

How do you monitor a model when ground truth labels are delayed?

Track proxy signals such as input and prediction drift, and estimate performance with a method like CBPE until labels arrive, then compare the estimate with realized performance. The estimate assumes calibrated probabilities and does not detect concept drift.

What is the difference between training-serving skew and drift?

In Google Cloud's definition, training-serving skew is a gap between the production feature distribution and the training data distribution. Drift is a change in the production distribution over time.

How often should a production model be retrained?

Retrain on evidence. A schedule works when labels arrive regularly, and measured degradation is the strongest trigger. Drift without degradation is a reason to investigate, and Vela and colleagues note that research still lacks a systematic answer on retraining triggers.