Machine Learning Operations (MLOps)

Definition

MLOps (Machine Learning Operations) is the engineering practice of getting machine learning systems into production and keeping them right — and it exists because a deployed model is built from three inputs, code, training data and trained weights, of which only the code lives in git. That single asymmetry generates almost everything on the MLOps tool list. Data versioning exists because you cannot commit a 50 GB table. Model registries exist because you cannot commit a 14 GB weights file. Drift monitoring exists because the data that arrives tomorrow is drawn from a world that is not under version control at all.

The consequence a reader should take away is that a machine learning service can be simultaneously healthy and wrong. A web service that returns HTTP 200 and passes its tests is working. A model that returns 200 and passes every unit test can still be silently producing worse answers every week, because its correctness depends on a data distribution that changed after the tests were written — and nothing in a normal alerting stack is looking at that. Sculley et al.'s Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) put the ratio bluntly: "a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code". MLOps is the discipline of making that 95% boring.

How It Works

The artifact with three inputs

Ordinary software has one input you must control: source code. Reproduce the commit, reproduce the binary. Machine learning has three, and they fail differently.

Code is the one git handles well. Data is the one it handles badly — a 50 GB training table committed weekly for a year is 52 × 50 GB = 2.6 TB of near-identical bytes: git stores files above core.bigFileThreshold (512 MiB by default) "deflated in packfiles, without attempting delta compression", so each week's near-duplicate is kept in full, and every clone drags the whole history down with it. Tools like DVC and lakeFS fix this by storing content hashes in git and the bytes elsewhere, chunked so that identical chunks are stored once. If only 2% of the rows change each week, the same year costs 50 GB for the first snapshot plus 51 × 1 GB for the deltas — about 101 GB, roughly 26× smaller, and the difference recurs on every monthly storage bill and every developer's laptop.

Weights are the input that has no analogue in software at all. A 7-billion-parameter model stored at bf16 is 7 × 10⁹ × 2 bytes = 14 GB for the weights alone. A resumable training checkpoint carries far more: fp32 weights, gradients and the two Adam moment tensors are 4 bytes each, so 16 bytes per parameter, or 112 GB per checkpoint. Save one every 1,000 steps of a 40,000-step run and a single training job leaves 4.48 TB behind. Keep weights only and it is 560 GB; keep the best three and it is 42 GB. That decision — what a model registry retains and for how long — is a real MLOps design choice with a four-figure monthly price attached, and it is the kind of question nobody has to ask about a JAR file.

The machinery, and what each piece is actually for

Once you accept that two of the three inputs cannot live in the repository, the standard MLOps stack stops looking like a vendor checklist and starts looking like a list of consequences:

• Experiment tracking (MLflow, Weights & Biases) records the tuple (code commit, data version, hyperparameters, resulting metrics) for every run — because without it "the model we shipped in March" is not a recoverable object, merely a file someone still has.
• A model registry gives a trained artifact a version, a stage and a lineage pointer back to that tuple, so rolling back means promoting a previous version rather than retraining and hoping.
• A feature store exists to make the same feature computation serve both training and inference. Uber built Michelangelo around one holding roughly 10,000 shared features precisely so that two teams could not implement "average trip length" two subtly different ways.
• Drift monitoring watches the inputs, not the outputs, because the outputs cannot be scored until labels arrive — which may be never.
• Continuous training is the stage with no DevOps equivalent: a pipeline that retrains, revalidates and redeploys on a schedule or on a drift trigger, without a human writing any new code.

Why monitoring is the hard part

In normal monitoring you compare a measurement against a threshold. In production machine learning you frequently cannot take the measurement. To compute accuracy you need the true label, and for a churn model the label arrives ninety days later; for a fraud model it arrives when the chargeback does; for a recommender predicting whether a user will book, it arrives only for the users who booked, so the predictions made for everyone else never get a label at all. Booking.com's engineers describe exactly that gap in 150 Successful Machine Learning Models (KDD 2019) and answer it with Response Distribution Analysis — a histogram of the model's own output scores, read for pathologies, on the reasoning that a healthy binary classifier's score distribution should be bimodal near 0 and 1, and a sudden smooth unimodal hump in the middle means something is wrong even though no label has arrived to prove it.

Real-World Applications

Instacart, March 2020, is the clearest published case of a model breaking without any code changing. As pandemic buying emptied shelves, the accuracy metric for its item-availability model "dropped to 61% from 93%", and the fix was operational rather than algorithmic: the team cut the training window from several weeks down to about ten days of data so the model would stop learning from a world that no longer existed, and increased scoring frequency from every three hours to hourly. Accuracy recovered to roughly 85%, still below the pre-pandemic level (Fortune, 9 June 2020). Nothing in that incident would have been caught by a test suite; it was caught by someone watching a metric that only exists if you built the pipeline to compute it.

Uber's Michelangelo is the canonical published feature-store deployment: a shared store of about 10,000 features, with the same computation path used for offline training and online serving, and — by Uber's own account of the platform — its highest-traffic models serving more than 250,000 predictions per second. The architectural point is the shared path — the platform's answer to training-serving skew is to make it structurally impossible to write the feature twice.

Booking.com measured something most teams only assume. Across roughly 150 RCT-validated models, they found that "an increase of about 30% in latency costs more than 0.5% in conversion rate", and — more uncomfortably — that "increasing the performance of a model does not necessarily translate to a gain in value". Both findings change what an MLOps pipeline should gate on: a benchmark improvement that ships a slower model can be a net loss, so the deployment gate has to include serving latency, not just offline metrics.

Regulation is now a forcing function. The EU AI Act requires that high-risk AI systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system" (Article 12), and its Annex IV technical documentation must cover "the training methodologies and techniques and the training data sets used", including "information about their provenance, scope and main characteristics". Those are reproducibility requirements written into law: an organisation that cannot rebuild last quarter's model from its recorded data version and code commit has a compliance problem, not just an engineering one.

The open-source stack that implements all this is worth naming because it is what teams actually install: MLflow and Weights & Biases for experiment tracking and registries, DVC for data versioning, Feast for feature serving, Kubeflow and Airflow for orchestration, Evidently and WhyLabs for drift reports — sitting on top of a training framework like PyTorch.

Key Concepts

• Training-serving skew: the feature values at inference differ from those at training. Google's Rules of Machine Learning gives the structural fix — "save the set of features used at serving time, and then pipe those features to a log to use them at training time" — which removes the second implementation rather than trying to keep two in sync.
• CACE: Sculley et al.'s "Changing Anything Changes Everything". Change the distribution of one input feature and the learned importance of all the others shifts, so there is no such thing as a local change to a model — which is why ML systems need end-to-end validation where software gets away with unit tests.
• Continuous training (CT): the retraining pipeline, distinct from CI and CD. It is the only part of the stack that ships a new artifact without a human writing a line of code.
• Point-in-time correctness: every feature in a training row must be computed as of the moment the prediction would have been made, not as of when the dataset was assembled. Get it wrong and you train on information from the future, and the offline metric looks wonderful.
• Shadow and canary deployment: run the new model on live traffic without acting on its output, or on 1% of it, because offline validation cannot tell you how a model behaves on the traffic it will actually change.

Challenges

Retraining cadence is arithmetic that most teams never do. Suppose a model trains on 8 GPUs for 4 hours — 32 GPU-hours a run. Nightly retraining is 30 runs, or 960 GPU-hours a month; at $2 per GPU-hour that is $1,920 a month. Weekly retraining is about 138 GPU-hours, or $275. Now price the decay. On a stream of 1,000,000 card transactions a month at a 0.15% fraud rate there are 1,500 fraudulent transactions; a model at 80% recall catches 1,200 and misses 300. Let staleness cost two points of recall — 78% catches 1,170 — and 30 more frauds get through each month. At an average loss of $120 each, those two points cost $3,600 a month. Nightly retraining clears its own bill about 1.9×, so you do it. Run the same model over 300,000 transactions a month and the recovered loss is $1,080 against the same $1,920, and nightly retraining is a straight loss: the break-even is around 530,000 transactions a month. The right cadence is a property of your traffic volume, not of anyone's maturity model.

Silent degradation is the failure mode with no alarm attached. In the fraud example above, the service's error rate is zero throughout. Latency is unchanged. Every unit test passes. The only symptom is 30 extra chargebacks a month, arriving weeks late and attributed to fraudsters getting cleverer. This is the concrete answer to "what breaks if I skip MLOps": not an outage, a slow leak that your existing observability stack is architecturally incapable of seeing.

Reproducibility fails at the seams, not the centre. Teams pin requirements.txt and then discover that the training set was assembled by a notebook querying a mutable warehouse table, that the random seed was set in one of three libraries, and that a GPU kernel changed between driver versions. The regulator's question — rebuild the model you were using in Q1 — is answerable only if the data version, code commit and environment were captured together at training time, because none of them can be reconstructed afterwards.

Two implementations of one feature is the bug that keeps recurring. A data scientist writes the feature in pandas over a whole table; an engineer rewrites it in Java inside a request handler. They agree on the happy path and disagree on nulls, time zones, or what "last 30 days" means. The model's offline evaluation is honest and its production behaviour is worse, with no error anywhere. The ## Code Example below is the cheapest version of this bug, and it is extremely common.

The organisational challenge is real but is not a talent shortage. It is that the three inputs have three different owners — data engineering owns the tables, data science owns the training code, platform owns the serving path — and drift in any one of them is invisible to the other two. Feature stores and registries are as much contracts between teams as they are software.

Future Trends

The largest change in MLOps since 2023 is that for a growing share of teams the artifact stopped being a model they train. When the system is a call to a hosted large language model, two of the three inputs vanish and are replaced by different ones — and the machinery is being rebuilt accordingly, under the name LLMOps.

• Prompt and model-string versioning replaces weight versioning. There are no 14 GB checkpoints to store; there is a prompt template, a model identifier and a temperature, all of which change system behaviour and none of which git treats as anything but text. The registry problem becomes trivially small and the evaluation problem becomes the whole job.
• Evals replace accuracy metrics. There is usually no labelled test set, so quality is measured by a rubric scored by humans or by another model — a judge whose own outputs vary run to run. A regression gate built on a noisy metric needs enough samples to distinguish a real quality drop from judge variance, which is a statistics problem the accuracy-metric era never had.
• Drift now arrives from the provider. Your input distribution may be perfectly stable while the endpoint behind gpt-* or claude-* is updated, deprecated or retired on a published schedule. Pinning a dated model snapshot is the LLMOps equivalent of pinning a dependency, and the pin expires.
• Cost monitoring moves from capex to per-request opex. The metric that matters is no longer GPU utilisation during training but tokens per request in production, where a prompt template edit that adds 400 tokens of instructions raises the bill on every call, forever, and shows up in no test.
• Non-determinism becomes a first-class operational fact. Temperature 0 is not a guarantee of identical output across batching and kernel changes, so "the same input produced a different answer" stops being a bug report and starts being a property the system must be designed to tolerate.

What does not change is the spine. Whether the artifact is a set of weights you trained or an endpoint you rent, the system's behaviour still depends on inputs that are not in your repository and not under your control, and the job is still to version what you can, watch what you cannot, and be able to answer what the system was doing last quarter.

Code Example

The commonest training-serving skew bug needs no machine learning to demonstrate. A feature is computed as of the label date during training and as of now during serving — so the model is scored on a range of values it never saw:

from datetime import date

# One user, one feature: "how long had this account existed when we scored it?"
signup = date(2025, 3, 14)

# TRAINING. The label — did this user churn? — is only known once the churn
# window closed, so the training row is built as of the label date.
label_date = date(2026, 1, 10)
account_age_at_training = (label_date - signup).days

# SERVING. The same feature, computed in the request handler, as of today.
scoring_date = date(2026, 7, 22)
account_age_at_serving = (scoring_date - signup).days

print(f"account_age_days at training time: {account_age_at_training}")
print(f"account_age_days at serving time:  {account_age_at_serving}")
print(f"shift: {account_age_at_serving - account_age_at_training} days "
      f"({account_age_at_serving / account_age_at_training:.2f}x)")

Output:

account_age_days at training time: 302
account_age_days at serving time:  495
shift: 193 days (1.64x)

Every account is 1.64× older at scoring time than the model was ever told accounts could be. Offline evaluation is clean, the deployment is green, and the model's ranking quality quietly falls. A feature store fixes this by making training rows read features through a point-in-time join against the same store that serves them — which is the entire reason that piece of infrastructure exists.