Model Monitoring vs Observability: What ML Startups Get Wrong

Most ML teams conflate model monitoring with infrastructure monitoring - and the difference matters. Infrastructure monitoring tells you whether your ML system is up, responding to requests, and using compute resources correctly. Model monitoring tells you whether your ML system is doing its job correctly. Both are necessary. Most companies have only the first.

The second conflation - monitoring vs. observability - matters too. Model monitoring is watching known metrics against known thresholds. ML observability is the capability to answer arbitrary questions about your model’s behaviour in production. Monitoring catches the failure modes you anticipated. Observability helps you diagnose failure modes you didn’t anticipate - which, in ML systems, are often the important ones.

Why Infrastructure Monitoring Isn’t Enough

Infrastructure monitoring for an ML system typically tracks: API latency, request throughput, error rates, GPU utilisation, memory usage, and uptime. These are important signals. They don’t tell you whether your model is making good predictions.

The failure modes that infrastructure monitoring misses:

Silent accuracy degradation. Your model’s prediction accuracy degrades over 6 weeks as production data distribution drifts from training data. Latency is fine. Error rate is 0%. The degradation is invisible until a user reports that the recommendations have gotten worse, or until a business metric starts declining.

Label shift. The base rate of the thing your model predicts changes in production. A fraud model trained on 2% fraud prevalence is now seeing 8% fraud prevalence. Infrastructure is healthy. The model is miscalibrated for the new distribution, and fraud is being missed.

Feature distribution shift. One of the features your model relies on starts returning null values more frequently, or the values in a key feature have shifted in distribution. The model is receiving different input than it was trained on. Infrastructure monitoring sees nothing.

Concept drift. The relationship between features and labels changes in production - fraud patterns evolve, customer behaviour shifts, external conditions change. The model learned the old relationship. It’s now making predictions based on a relationship that no longer holds.

None of these produce infrastructure alerts. All of them cause real harm.

What to Monitor: The Three Drift Types

Data drift (also called input drift or covariate shift) is the condition where the distribution of features the model receives at serving time differs from the distribution at training time. Monitoring for data drift requires:

• Computing feature distributions over a sliding window of production requests
• Comparing these distributions to the training distribution using statistical tests (PSI - Population Stability Index - is the most widely used in practice)
• Alerting when drift exceeds a threshold that has historically preceded accuracy degradation

The challenge: data drift doesn’t always cause accuracy degradation (if the model has learned a robust relationship that holds across distributions, drift may not matter), and the threshold that triggers alerting requires calibration. Monitoring data drift without connecting it to model accuracy creates alert fatigue.

Prediction drift (also called output drift) is the condition where the distribution of the model’s predictions changes. A regression model whose prediction distribution shifts significantly is almost certainly seeing something different from what it was trained on. A classification model whose positive rate doubles is worth investigating.

Prediction drift is easier to monitor than data drift - you don’t need labels, and predictions are always available - and it’s a reliable leading indicator of problems even when you don’t have ground truth to measure accuracy directly.

Concept drift (also called label drift or posterior drift) is the condition where the relationship between features and labels changes. This is the hardest drift to detect without ground truth, because by definition it requires knowing the actual labels for production predictions. Strategies include: delayed label collection (comparing predictions to eventual outcomes), proxy metrics that correlate with accuracy, and cohort analysis that compares model performance across time periods.

The Tools: What Works and When

Evidently AI is the most accessible open-source tool for ML monitoring. It generates reports for data drift, prediction drift, and data quality, and can be run as a standalone monitoring job against batch data or integrated into a real-time monitoring pipeline. Good default choice for teams starting with ML monitoring - low setup friction, good reports, free.

Arize AI is a managed ML observability platform. Beyond drift monitoring, Arize provides embedding drift monitoring (critical for LLM and vector-based applications), slice performance analysis, and a UI for investigating production model behaviour. The observability positioning is meaningful - you can ask “why did my model underperform on this customer segment last Tuesday?” and get an answer. More expensive than open-source, justified at production scale.

WhyLabs is positioned similarly to Arize - managed ML observability with a strong focus on data quality and drift monitoring. Good integration with common ML frameworks.

Cloud-native options (Vertex AI Model Monitoring, SageMaker Model Monitor) are the right default if you’re already on those cloud platforms and want minimal integration overhead. Less flexible than specialised tools, sufficient for standard drift monitoring use cases.

The Metrics That Actually Matter

The monitoring metrics that reflect real model health - as opposed to metrics that look good in a dashboard without telling you anything actionable:

Ground truth accuracy (where available) is always the gold standard. If you can collect labels for production predictions - even with a delay - tracking accuracy over time is the most reliable model health signal.

Business outcome metrics correlated with model quality. For a recommendation model, downstream conversion rate. For a fraud model, fraud loss rate. For a credit model, charge-off rate. These are lagging indicators but the most meaningful signal of whether the model is doing what it’s supposed to do.

Prediction distribution by cohort. Aggregate prediction distribution monitoring misses subgroup performance issues. Monitoring performance by customer segment, geography, product category, or other relevant dimensions catches localised degradation that aggregate metrics don’t reveal.

Feature null rates and range violations. Simple data quality checks - null rates for critical features, out-of-range values for numerical features - are cheap to implement and catch upstream data pipeline failures that corrupt model inputs.

The goal of ML observability is not to generate alerts - it’s to give you the information you need to answer questions about model behaviour when something goes wrong. Build the instrumentation that makes those questions answerable.

Talk to us about your ML architecture review and we’ll assess your monitoring and observability gaps alongside your infrastructure.