Table of Contents
- The Question That Exposes Everything
- Four Things That Must Be Versioned Together
- Why Notebooks Do Not Survive Production
- What a Model Registry Actually Buys You
- Training-Serving Skew
- Deployment Patterns for Models
- A Minimum Viable MLOps Stack
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Reproducibility requires versioning code, data, configuration, and environment together. Versioning only code — which is what most teams do — reproduces nothing, because the model was a function of all four.
The Question That Exposes Everything
Ask a team to rebuild the model currently serving production traffic, from scratch, and produce identical predictions.
Most cannot. The training script has changed since. The dataset has grown. The library versions have moved. The hyperparameters were passed on a command line nobody recorded. The preprocessing had a step someone applied manually. The notebook was overwritten.
The model works. Nobody can rebuild it. This is a fragile position for several reasons that become urgent at inconvenient moments.
You cannot debug what you cannot reproduce. When the model starts behaving oddly, distinguishing a model problem from a data problem requires reconstructing the original conditions. You cannot improve it safely, since any change lacks a comparable baseline. You cannot roll back reliably, because “the previous model” is a file whose provenance is unclear. And you cannot answer auditors — in regulated contexts, “what data trained this decision system?” is a question with legal weight.
The gap between software engineering and machine learning engineering is precisely here. Code is versioned by default, because that is what version control does. Models depend on code plus data plus configuration plus environment, and only one of those four is versioned by default.
Four Things That Must Be Versioned Together
Reproducibility means recording every input to the training process, linked by a single identifier.
Code. Training script, preprocessing, feature engineering, evaluation. Version control handles this, provided the model records which commit produced it.
Data. The exact dataset used. Not “the customer table” but its state at a specific point. Options include immutable snapshots, content-addressed storage, or a query plus timestamp against a versioned table format. The critical property is that the same identifier retrieves the same rows in a year.
Configuration. Hyperparameters, random seeds, feature selections, split definitions. These belong in a committed file rather than command-line arguments or notebook cells, because arguments are not recorded and cells get edited.
Environment. Library versions, and for GPU training the driver and framework versions too. Numerical results differ across library versions in ways that are small individually and compound. A lockfile or container image digest is the reliable answer.
Linked together, this becomes a manifest:
model_id: churn-predictor-v14
git_commit: 3f8a91c
dataset:
uri: s3://data/churn/snapshot-2026-07-15/
sha256: b41d9e...
rows: 1284391
config:
learning_rate: 0.001
max_depth: 8
random_seed: 42
environment:
image: ml-training:2026.07.1@sha256:9c2f...
metrics:
auc: 0.874
precision_at_10: 0.31
trained_at: 2026-07-16T09:22:11Z
trained_by: ci-pipeline-run-4471Any model in production should have this manifest retrievable by its identifier. If it does not, the model is unreproducible regardless of how good its metrics were.
Why Notebooks Do Not Survive Production
Notebooks are excellent for exploration and structurally unsuited to production training, for reasons inherent to the format rather than to discipline.
Execution order is not the document order. Cells can run in any sequence, and the visible notebook does not record which sequence produced the output. A notebook that appears to work may depend on a cell that has since been edited.
Hidden state accumulates. Variables persist from deleted cells. A notebook can work in a session and fail on a fresh run, and the failure will surprise its author.
They resist review and testing. JSON with embedded output produces unreadable diffs. Functions inside cells are not importable, so they cannot be unit tested.
Manual steps go unrecorded. The cell that was run once to fix a data issue, then deleted, is now an invisible part of the pipeline.
The productive resolution is not banning notebooks. It is treating them as the exploration medium and extracting the resulting logic into modules that a pipeline invokes. Notebooks import from those modules for interactive work, so exploration and production share the same code path rather than diverging.
The practical test: can your training run from a clean checkout with a single command, without a human executing anything by hand? Until that is true, your training process contains steps nobody has recorded.
What a Model Registry Actually Buys You
A model registry is a catalogue of trained models with their metadata, lineage, and lifecycle stage. It sounds like bureaucracy and solves several concrete operational problems.
It makes the deployed version unambiguous. Rather than a file path someone remembers, production references a registry entry with a version number. “What is running?” becomes a query.
It preserves lineage. Every model links to the code commit, dataset version, and configuration that produced it. This is what makes debugging and auditing possible at all.
It enables comparison. Metrics stored alongside each version make “is the new model better?” answerable from records rather than recollection.
It makes rollback trivial. Promoting a previous version is an operation rather than an archaeology project.
It creates a promotion gate. Models move through stages — development, staging, production, archived — with checks required to advance. This is where fairness evaluation, latency verification, and approval requirements attach.
The minimum useful implementation is modest: a table recording model identifier, version, artefact location, the four version references, evaluation metrics, current stage, and who promoted it when. Dedicated tooling adds convenience and the discipline matters more than the tool.
Training-Serving Skew
The most common cause of a model performing worse in production than in evaluation, and the most preventable.
Skew occurs when features are computed differently during training and serving. Typically training uses a batch pipeline over historical data written in one language, while serving computes features on the fly in another. Two implementations of the same logic diverge, quietly.
Specific recurring causes:
Different code paths. A subtle difference in null handling, rounding, or normalisation produces feature values the model never saw.
Time leakage. A feature computed from the full historical dataset includes information unavailable at prediction time. Aggregations are the usual culprit — an average computed over all data leaks the future.
Missing-value handling differences. Training filled nulls with the column mean; serving passes zero.
Category encoding drift. Categorical values encoded by index will shift if the vocabulary is rebuilt in a different order.
The structural fix is computing features in one place used by both paths. A feature store provides this: define the transformation once, serve it to training as historical values and to inference as current values, guaranteeing consistency by construction.
Where a feature store is too heavy, a lighter version works — a shared library implementing every transformation, imported by both the training pipeline and the serving code. The essential property is a single implementation, not a particular product.
A useful detection method: log the feature vectors your serving path computes, and compare their distributions against the training set. Skew shows up as a distribution mismatch and is otherwise nearly invisible.
Deployment Patterns for Models
Model deployment carries risk that ordinary code deployment does not, because a model can be syntactically fine and behaviourally wrong.
Shadow deployment. Run the new model alongside the current one, serving the old predictions while logging both. Compare on real traffic with no user impact. This is the safest way to validate on production data and requires no user exposure.
Canary release. Route a small traffic percentage to the new model, monitor business metrics rather than only technical ones, and expand gradually. Business metrics matter here because a model can improve offline metrics while harming outcomes.
Blue-green. Two complete environments, switch traffic, keep the old one warm for immediate rollback. Simple and doubles serving cost during the transition.
Champion-challenger. Continuous comparison where a challenger model runs in shadow indefinitely and is promoted when it demonstrably outperforms over a sustained period. This suits environments where drift is constant.
Shadow deployment deserves emphasis because it is underused relative to its value. It catches training-serving skew, latency problems, and unexpected input handling before any user sees the new model — and the only cost is the compute to run inference twice.
A Minimum Viable MLOps Stack
For a team without existing infrastructure, ordered by return on effort:
| Priority | Capability | Minimum implementation |
|---|---|---|
| 1 | Experiment tracking | Log params, metrics, artefacts per run |
| 2 | Data versioning | Immutable snapshots with content hashes |
| 3 | Pipeline as code | One command trains from clean checkout |
| 4 | Model registry | Table of versions with lineage and stage |
| 5 | Shadow deployment | Log both models’ predictions |
| 6 | Drift monitoring | Track feature and prediction distributions |
| 7 | Feature consistency | Shared transformation library |
| 8 | Automated retraining | Scheduled or drift-triggered |
The first three are where most of the value sits and are achievable in days rather than quarters. Experiment tracking alone eliminates the most common frustration in applied machine learning — not knowing which configuration produced the good result.
Resist building the full platform first. Teams that begin with orchestration frameworks and feature stores before they have reproducible training have built infrastructure around a process that does not work yet.
Common Pitfalls
Versioning code but not data. The model is a function of both. Half the inputs recorded reproduces nothing.
Training in notebooks without extraction. Hidden state and unrecorded execution order make reproduction impossible in principle.
Separate feature implementations for training and serving. Guarantees eventual skew.
No baseline comparison. Without the current model’s metrics on the same evaluation set, “better” is an assertion.
Deploying without shadow validation. Offline metrics do not capture skew, latency, or unexpected inputs.
Unversioned environments. Library upgrades change numerical results in ways that accumulate silently.
Building the platform before the pipeline. Orchestration around an unreproducible process automates the wrong thing.
Conclusion
The distinguishing characteristic of a mature machine learning practice is that any model in production can be rebuilt from recorded inputs and produce the same predictions. That property is what makes debugging, improvement, rollback, and audit possible, and it requires versioning four things together rather than one.
Start with experiment tracking, because it costs almost nothing and immediately answers which configuration worked. Add immutable data snapshots referenced by hash. Get training running from a clean checkout with one command. Those three steps convert an unreproducible process into a reproducible one, and everything else — registry, shadow deployment, drift monitoring, automated retraining — builds naturally on top.
The alternative is the position most teams occupy: a model that works, that nobody can rebuild, awaiting the day someone needs to explain it.
Frequently Asked Questions
Do small teams need a model registry? Some form of one, yes, though a spreadsheet or database table suffices initially. The requirement is knowing what is deployed and what produced it, not a particular product.
How do you version large datasets without copying them repeatedly? Content-addressed storage with deduplication, or versioned table formats supporting time-travel queries. Both let you reference an exact state without duplicating unchanged data.
Is a feature store necessary? Not for a single model with simple features. It becomes valuable with several models sharing features, or when training and serving paths would otherwise be implemented separately. The consistency guarantee is the point, not the technology.
How often should models be retrained? Determined by measured drift rather than a fixed schedule where possible. Absent drift monitoring, monthly to quarterly is a reasonable default, with immediate retraining after known upstream changes.
What is the difference between model drift and data drift? Data drift is the input distribution changing. Concept drift is the relationship between inputs and outcomes changing. Both degrade performance; data drift is detectable without labels, concept drift generally is not.
Should training pipelines run in CI? Yes for the validation path — a short training run on a data sample verifying the pipeline executes. Full training is usually too expensive for every commit and belongs on a schedule or trigger.
How do you handle model rollback when the schema changed? This is why models must record their expected input schema. Rolling back a model requires rolling back the feature computation it expected, which means feature transformations need versioning alongside the model.
