Key takeaway: A production model degrades silently because the metric that would reveal it depends on labels that lag by weeks. Monitor the inputs and outputs, which need no labels at all.
Why Accuracy Is Not Enough
A credit model’s true outcome is known when a loan defaults or matures — months later. A churn model’s label arrives at the end of the prediction window. A fraud model gets labels only for cases someone investigated, which is a biased subset.
If accuracy is your only monitor, you learn about degradation long after it started affecting decisions.
The Layers Worth Instrumenting
Input distribution. Compare live feature values against the training distribution. A numeric feature whose mean shifts, a categorical feature with new unseen values, a null rate that jumps — all detectable within minutes and all predictive of degraded performance.
Prediction distribution. If a model that predicted positive 4% of the time begins predicting 11%, something changed even without a single label. This is the cheapest useful monitor available and is frequently absent.
Feature importance stability. A meaningful shift in which features drive predictions suggests the relationship the model learned no longer holds.
Business metrics. Downstream outcomes — conversion, approval rate, ticket volume — often move before labels arrive and connect model behaviour to consequences.
| Signal | Label needed | Detection lag |
|---|---|---|
| Input drift | No | Minutes |
| Prediction drift | No | Minutes |
| Null rate and schema | No | Immediate |
| Business proxy metrics | No | Hours to days |
| Accuracy | Yes | Days to months |
Distinguishing Two Different Problems
Data drift means the inputs changed while the underlying relationship held. A new marketing channel brings users with different characteristics; the model may still be correct for them.
Concept drift means the relationship itself changed. What predicted fraud last year does not now, because attacker behaviour adapted.
The distinction determines the response. Data drift may need nothing more than confirming the model generalises to the new population. Concept drift requires retraining on recent data, and retraining on old data will not help.
Input monitoring detects data drift. Only labels or business outcomes reveal concept drift, which is why proxy metrics matter so much.
Avoiding Alert Fatigue
Statistical drift tests on many features generate constant alerts, because with enough features something is always drifting at p < 0.05. That noise trains people to ignore the dashboard.
Alert on features the model actually depends on, weighted by importance. Use effect size — population stability index or Wasserstein distance — rather than significance, since significance scales with sample size and becomes meaningless at production volume. Require a shift to persist across several windows before paging, so a single anomalous batch does not wake anyone.
Schema violations and sudden null-rate changes are the exception. Those indicate a broken upstream pipeline and should alert immediately, because they usually mean the model is receiving garbage.
The Bottom Line
Monitor input distributions, prediction distributions and business proxies so degradation is visible before labels arrive. Alert on effect size for important features rather than p-values across all of them, and treat schema and null-rate breaks as immediate pipeline failures.
