Monitoring

Definition

Monitoring a machine learning system is the practice of watching a program whose code has not changed for evidence that the world it was trained on has. That is the whole difference from ordinary application monitoring, and it produces a failure mode normal observability is architecturally incapable of seeing: a model that returns HTTP 200 well inside its latency budget, passes every unit test, keeps its error rate at zero — and is quietly wrong more often each week.

The asymmetry that organises everything else is this: you can see the inputs move today, and you cannot see the accuracy fall until the labels arrive. For a card-fraud model, the label is a chargeback, and US billing-error rules give a cardholder 60 days from the statement to file one (12 CFR 1026.13(b)(1)). On a calendar-month billing cycle, a transaction on 22 July lands on a statement transmitted 1 August, and the dispute window closes 30 September — so whether today's predictions were right is a fact you will learn in 70 days. Every technique in this page is a way of coping with that gap.

How It Works

Application monitoring always gets an answer; model monitoring usually does not

A web service tells you when it fails. The request returns a 500, or times out, or the queue backs up. The measurement is cheap and immediate because success is a property of the response itself.

A model returns 0.83. Nothing in that response says whether 0.83 was the right answer, and nothing ever will until reality reports back. So monitoring splits into two layers that get confused constantly:

• The serving layer — latency, throughput, GPU utilisation, error rates, queue depth. This is ordinary inference infrastructure monitoring, it is a solved problem, and Prometheus and Grafana handle it. It tells you nothing about correctness.
• The model layer — is this thing still right? No standard web stack measures this, and in production you usually cannot measure it directly at all.

The rest of monitoring is about the second layer.

Three things drift, and only two of them are visible

The vendor taxonomies of "monitoring types" are mostly marketing, but one distinction is real and mechanical, because it is a statement about which probability distribution moved.

Data drift (also called covariate shift) is a change in P(x), the distribution of the inputs. Say a credit model built on applicants with a median income of £34,000 starts receiving applicants at £48,000. You can detect this today, from the requests alone, with no labels whatsoever — which is exactly why input monitoring is the backbone of every practical setup.

Prediction drift is a change in P(ŷ), the distribution of the model's own output scores. Also observable immediately. It is partly downstream of data drift, but it earns its own place because it catches things per-feature checks miss: forty features can each shift within tolerance while their combination pushes the score distribution somewhere new.

Concept drift is a change in P(y|x) — the relationship between input and correct answer. The inputs can be pixel-for-pixel identical to last year's and the right answer has changed anyway. Imagine a delivery-time model after a new regional warehouse opens: the orders arriving look exactly as they always did, so every input monitor stays flat, while the true delivery time for those same inputs has dropped by a day, so every estimate the model produces is now a day too long. Concept drift cannot be detected from inputs. Ever. It is the drift that costs money, and it is invisible.

That asymmetry — cheap detection for the harmless case, no detection for the expensive one — is why ML monitoring is a discipline rather than a dashboard.

Measuring a shift you can actually see

The standard instrument for input drift is the Population Stability Index (PSI). Bin a feature using the training data's own quantiles, then compare the share of traffic in each bin now against the share then:

PSI = Σ (aᵢ − eᵢ) · ln(aᵢ / eᵢ)

Do one by hand. Cut transaction amount at the training quartiles, so the reference is 25% per bin by construction. Tuesday's traffic comes in at 18% / 22% / 28% / 32% — visibly lopsided on a bar chart. Each term is a difference times a log ratio: (0.18−0.25)·ln(0.72) = 0.0230, then 0.0038, 0.0034, 0.0173. PSI = 0.048. Under every conventional threshold. The chart looked alarming and the statistic says nothing has happened.

Friday comes in at 10% / 15% / 30% / 45%, and the same four terms sum to 0.315. That is a real shift, and PSI is far less twitchy than the human eye — which is most of the reason to use it.

It is worth knowing what PSI is, because the name hides it: expand the formula and it is exactly KL(a‖e) + KL(e‖a), the symmetrised Kullback–Leibler divergence, measured in nats. For the Friday window those two halves are 0.1509 and 0.1643, and they sum to 0.3152. Nothing exotic is happening — "population stability index" is credit-scoring vocabulary for an information-theoretic quantity that predates it by decades.

The number nobody tells you: what PSI reads when nothing is wrong

Almost every article quotes the same rule of thumb — below 0.10 "little change", 0.10 to 0.25 "moderate change", 0.25 and above "significant change, action required". Yurdakul and Naranjo, in Statistical properties of the population stability index (Journal of Risk Model Validation, 2020), quote that rule and then say the quiet part: "These benchmarks are used without reference to statistical type I or type II error rates."

They then derive the property that matters operationally. If the two samples really are drawn from the same distribution, (1/n + 1/m)⁻¹ · PSI is approximately chi-square with B − 1 degrees of freedom, where B is the number of bins. Rearranged, the expected PSI between two windows when nothing has changed is roughly

E[PSI] ≈ (B − 1) · (1/n + 1/m)

Put numbers in it. Ten bins, two windows of 1,000 rows each: 9 × 0.002 = 0.018. Fine. Ten bins, two windows of 200 rows each: 9 × 0.01 = 0.090 — which is essentially the 0.10 "moderate change" line, produced entirely by sampling noise, on data that has not moved at all. Simulate it and the theory holds to three digits — the code example below does exactly that. A daily drift check on a low-traffic segment, using the textbook threshold, is not monitoring; it is a random number generator wired to a pager.

The fix is not a better threshold, it is a threshold derived from your own window size. Comparing against a large, fixed reference set rather than against another small window drops the reference's term out of 1/n + 1/m and halves the noise floor; setting the alert at the 99th percentile of the chi-square then gives you a false-alarm rate you chose on purpose rather than inherited from a blog post.

Coping with labels that arrive late, or never

Return to the fraud model. Retraining on 1 September, the most recent month whose chargebacks have fully settled is June — July's dispute window does not close until 30 September. So the "accuracy" number on the dashboard on 1 September describes a model scoring June's traffic, and the decision to retrain is being made on zero percent of the last two months' labels.

Partial labels are available sooner, and they are treacherous rather than useless: the disputes that arrive in week one are not a random sample of the disputes that will eventually arrive. Large fraudulent charges get noticed faster than small ones, so an early-window "precision" is measured on a biased subset and reads systematically better than the truth. Treat it as a leading indicator with a known sign of error, not as the metric.

Three practices fill the gap, and all three are label-free:

• Watch the output distribution, not just the inputs. Booking.com's engineers formalised this as Response Distribution Analysis in 150 Successful Machine Learning Models (KDD 2019) — read the histogram of the model's own scores for pathologies, on the reasoning that a healthy binary classifier's scores are bimodal near 0 and 1, so a sudden smooth hump in the middle is a symptom even with no label in sight. The MLOps page covers the retraining lifecycle this feeds.
• Instrument the feature pipeline itself: null rate, out-of-range rate, cardinality, freshness per feature. Most "model degradation" incidents are really upstream data incidents, and these catch them in minutes rather than months.
• Find a proxy that settles fast. For a recommender, click-through today stands in for conversion next week; for a churn model, a support-ticket rate moves before the cancellation does. A proxy is not the objective, and treating it as one is its own failure — but a noisy signal today beats a clean signal in October.

Real-World Applications

Google Flu Trends is the canonical published case of a model that decayed in plain sight. It estimated influenza-like illness from search volume, and it worked well enough to be held up as the exemplar of big data. Then, as Lazer, Kennedy, King and Vespignani document in The Parable of Google Flu (Science, 14 March 2014), Nature reported in February 2013 that GFT was "predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention". This was not a spike. The paper shows GFT "missed high for 100 out of 108 weeks starting with August 2011", and that by then "even 3-week-old CDC data do a better job of projecting current flu prevalence than GFT".

The monitoring lessons are unusually clean. First, the cause was drift in a dependency nobody controlled: the authors note that "the official Google search blog reported 86 changes in June and July 2012 alone", so the search behaviour generating the features was being reshaped continuously by a system with its own commercial goals. Second — and this is the part worth internalising — the failure was only detectable because the CDC published independent ground truth on a two-week lag. The slow, inconvenient, old-fashioned label stream is the only reason anyone found out. Teams that switch off their expensive label collection because "the model is working" are switching off the instrument that would tell them otherwise.

The Epic Sepsis Model shows what happens when nobody measures at all. ESM is a proprietary early-warning model "implemented at hundreds of US hospitals". In External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients (JAMA Internal Medicine, August 2021), Wong and colleagues scored it against 38,455 hospitalisations at Michigan Medicine, of which 2,552 (7%) met the sepsis definition. The model's area under the ROC curve was 0.63 (95% CI 0.62–0.64) — against the "AUC, 0.76-0.83" that the paper says Epic Systems reported in internal documentation. It missed 1,709 of the 2,552 sepsis cases (67%) while alerting on 6,971 of all 38,455 hospitalisations (18%).

Do the division the paper implies: 2,552 − 1,709 = 843 sepsis cases detected, from 6,971 alerts. That is a precision of 12% — roughly seven alerts without sepsis for every alert with it, which is exactly the positive predictive value and the corrected number-needed-to-evaluate of 8 that the paper's Table 2 reports. The model had been running in hospitals for years. It took an academic health system with its own labelled outcomes to find out. A vendor's reported AUC is a claim about the vendor's population; it is not a monitor.

What teams actually install falls along the same seam as the two layers above. Prometheus and Grafana, or a cloud equivalent such as CloudWatch, cover the serving layer. Drift and data-quality reports come from Evidently, WhyLabs, Arize or Fiddler, or from the managed services built into SageMaker and Vertex AI. For LLM systems the unit of observation is a trace rather than a metric, and LangSmith and Langfuse record prompt, response, token counts and latency per call. The split is not cosmetic: the first tells you the service is up, the second tells you the inputs still look like the training set, and the third is the only place a wrong-but-fluent answer leaves a record.

Key Concepts

• Reference window: the "normal" everything is compared against. Comparing to the training set catches everything but alarms forever after the first legitimate shift; comparing to last week catches sudden breaks and is structurally blind to slow ones. This choice, not the threshold, decides which failures you can see.
• Label delay: the lag between a prediction and its ground truth. It ranges from seconds (ad click) to a quarter (chargeback, churn) to never (the loan you declined has no repayment history). It sets an upper bound on how fast any accuracy-based alarm can possibly fire.
• Proxy metric: a fast-settling stand-in for the real objective, used because the real objective is unobservable today. Useful precisely to the extent that you remember it is a proxy.
• Alert precision: of everything that fires, the share that corresponds to something worth doing. You can estimate it in advance from your test count and false-positive rate, and it predicts whether anyone will still be reading the channel in six months.
• Segment monitoring: aggregate metrics conceal subgroup collapse. Overall accuracy can hold steady at 91% while performance on a segment that is 4% of traffic falls off a cliff — and that segment is usually where the reputational and regulatory risk lives.

Challenges

Alert fatigue is arithmetic, not a personality flaw. Suppose you run a drift test on 40 features once a day, each tuned to a 1% false-positive rate. The expected number of false alarms per day is 40 × 0.01 = 0.4, and the probability that at least one fires is 1 − 0.99⁴⁰ = 33%. A third of all days carry a spurious alert. Over a quarter that is about 36 false alarms; if genuinely actionable drift occurs, say, once a quarter, alert precision is 1/37 = 2.7%. Muting the channel is the rational response to a channel that is 97% noise. Fixing it means correcting for the number of tests — one false alarm a month across 40 daily checks needs a per-test rate of 1/(40 × 30) = 0.00083 — and then accepting that you have traded away sensitivity to small real shifts, which is a decision to make deliberately rather than discover later.

The reference window has no correct answer. Take a feature drifting linearly, the bottom of four bins shedding 0.3 percentage points of mass a week and the top gaining the same. Week-over-week the PSI is 0.000125 — nothing, forever. After 52 weeks the distribution has gone from 25/25/25/25 to 9/20/30/41 and the PSI against the original reference is 0.250, right on the "action required" line. Same feature, same drift, two windows: one check would never have fired, the other fires exactly once. Seasonality makes it worse, because a retail model compared against last month alarms every December for reasons that are not a defect.

The loud failures and the quiet ones need different instruments. A feature pipeline that starts emitting nulls, which the serving code helpfully imputes as 0.0, is loud: if days_since_login collapses from ten populated bins into a single zero bin, PSI (with a 10⁻⁶ floor on empty bins) comes out near 12.4, some fifty times the 0.25 threshold. Input monitoring catches that within a cycle, and the model meanwhile is confidently telling you every user logged in today. The quiet failure is concept drift, which produces no input signal at all and shows up only when the labels land. Building only the first kind of monitor and believing you are covered is the commonest mistake in this area.

Monitoring is not free, and its cost scales with traffic rather than with model size. Logging 40 float64 features per request is 320 bytes; at 1,000 requests per second that is 86.4 million rows and 27.6 GB a day, about 10 TB a year before indexes or replication. Teams respond by sampling — 1% brings it to 276 MB a day — which is correct until you need to debug a rare segment that sampling threw away, or until your 200-row windows put you back at the noise floor calculated above. There is no configuration that is cheap, complete and statistically sound at once.

Generative systems have no accuracy metric to degrade. For a classifier, monitoring eventually gets a number. For a system that writes text, there is no label arriving in October either; quality is scored by a rubric, by humans, or by another model acting as judge. A judge is itself a non-deterministic component whose scores move run to run, so a quality regression has to be distinguished from judge variance, which needs enough samples to be a statistics problem rather than a glance at a dashboard. Meanwhile the failure that matters — a fluent, confident, wrong answer — produces a perfectly normal 200 response of perfectly normal length.

Future Trends

Monitoring is moving from the weights to the provider. When the model is a hosted large language model endpoint, there is nothing on your side to drift: no training set, no weights, no feature pipeline you own. What can change is the thing behind the URL. Google Flu Trends failed partly because Google changed its own search product 86 times in two months while GFT's features depended on it — the same structure, twenty years earlier. So the monitored objects become the model identifier (pin a dated snapshot, not a floating alias), the provider's deprecation schedule, and a fixed evaluation set replayed on a cadence with its scores stored as a time series. That last one is the closest thing to a drift detector available when the weights are not yours: if the eval score moves and your prompt did not, the endpoint did.

Three other shifts are already visible in what teams build:

• Drift detection is moving into embedding space. PSI over token frequencies is close to meaningless for free text. The workable substitute is to embed each request and monitor the distribution of distances to the reference set's centroids — the same statistic, applied to a learned representation rather than a raw feature. See embedding.
• The judge becomes a monitored dependency. If an LLM scores your outputs nightly, its own version, prompt and temperature belong in the change log next to the model's, because a quality "regression" that coincides with a judge upgrade is a measurement artefact.
• Cost per request joins correctness as a first-class metric. A prompt template edit that adds 400 tokens of instructions raises the bill on every call forever, breaks nothing, fails no test, and appears in no accuracy metric. It is a monitoring problem that did not exist when the model was a binary on your own hardware.

Code Example

The block below does three things: one PSI by hand on four bins, a check that PSI really is the symmetrised KL divergence, and the part that changes practice — a simulation of what PSI reads when two windows are drawn from the same distribution, against the chi-square prediction.

import numpy as np

rng = np.random.default_rng(0)

def psi(reference, window):
    """Population Stability Index between two binned distributions, in nats."""
    return float(np.sum((window - reference) * np.log(window / reference)))

# Four bins of "transaction amount", cut at the training-set quartiles, so the
# reference is 25% per bin by construction. Two later days, same four bins.
reference = np.array([0.25, 0.25, 0.25, 0.25])
tuesday   = np.array([0.18, 0.22, 0.28, 0.32])
friday    = np.array([0.10, 0.15, 0.30, 0.45])

print(f"Tuesday PSI = {psi(reference, tuesday):.4f}")
print(f"Friday  PSI = {psi(reference, friday):.4f}")

# PSI is exactly KL(a||b) + KL(b||a) — a symmetrised Kullback-Leibler divergence.
kl = lambda p, q: float(np.sum(p * np.log(p / q)))
print(f"KL(friday||ref) {kl(friday, reference):.4f} + "
      f"KL(ref||friday) {kl(reference, friday):.4f} = "
      f"{kl(friday, reference) + kl(reference, friday):.4f}")

# Now the question the thresholds never answer: what does PSI read when
# NOTHING has changed? Two windows of n rows and B = 10 bins, both drawn from
# the same distribution. Yurdakul & Naranjo: (1/n + 1/m)^-1 * PSI is
# approximately chi-square with B - 1 degrees of freedom.
CHI2_99_9DF = 21.666
print()
for n in (200, 1_000, 10_000):
    p = [0.1] * 10
    noise = [psi(rng.multinomial(n, p) / n, rng.multinomial(n, p) / n)
             for _ in range(4_000)]
    print(f"n={n:>6}  simulated mean {np.mean(noise):.4f}  99th pct "
          f"{np.quantile(noise, 0.99):.4f}   predicted mean {2 * 9 / n:.4f}  "
          f"99th pct {CHI2_99_9DF * 2 / n:.4f}")

Output:

Tuesday PSI = 0.0475
Friday  PSI = 0.3152
KL(friday||ref) 0.1509 + KL(ref||friday) 0.1643 = 0.3152

n=   200  simulated mean 0.0917  99th pct 0.2229   predicted mean 0.0900  99th pct 0.2167
n=  1000  simulated mean 0.0182  99th pct 0.0434   predicted mean 0.0180  99th pct 0.0433
n= 10000  simulated mean 0.0018  99th pct 0.0042   predicted mean 0.0018  99th pct 0.0043

The theory and the simulation agree to three digits, and the last block is the practical payload. At 200 rows per window the average PSI between two identical distributions is 0.092 and one day in a hundred exceeds 0.223 — so the standard 0.10 and 0.25 thresholds fire constantly on nothing. At 10,000 rows the same thresholds are so far above the noise that a real 0.05 shift would never be reported. A drift threshold that is not a function of your window size is not a threshold, it is a superstition, and the fix is one line: set the alert at CHI2_99 × (1/n + 1/m) for the bin count you actually use, and you know your own false-alarm rate.