Start with the data pipeline. The data pipeline is the production system; the model is an artifact. A team that has a data pipeline and a model in production is a team that has the foundation of MLOps. The maturity model is the path from manual (model trained in a notebook, served by hand) to full MLOps (model versioned, evaluated, deployed, monitored automatically). Pick the level you need and iterate.
Machine Learning Operations
The model is an artifact with a lifecycle. The data pipeline is the production system, not the model. The model evaluation is the per-request trust algorithm. The operating model is what keeps the ML system current with the data and the users.
On this page
§ 01 ·
What MLOps is
MLOps is the discipline of running machine learning systems as systems, not as experiments. The discipline covers four layers:
- Data pipeline. The production system that produces the training data and serves the inference.
- Model lifecycle. Training, evaluation, deployment, monitoring, retirement.
- Model evaluation. The per-request trust algorithm that determines whether the model should be used.
- Operating model. The team, the process, the tooling that keeps the system current.
The data pipeline is the production system
The most important property of an MLOps practice is that the data pipeline is the production system, not the model. The model is an artifact produced by the data pipeline; the data pipeline is the system that runs in production. An MLOps practice that focuses on the model and ignores the data pipeline is an MLOps practice that has the wrong object of attention.
The two failure modes
The two failure modes that come up most often are:
- Model without data pipeline. The model is deployed and the data pipeline is not. The model is making decisions on stale or wrong data.
- Model monitored for technical metrics only. The model is monitored for latency and throughput but not for the business metrics that the model is supposed to affect.
The right answer is to design the data pipeline and the model evaluation as part of the ML system, not as add-ons.
§ 02 ·
The architecture
The architectural primitives that have held up:
The data pipeline is the production system
The data pipeline is what runs in production. The model is a versioned artifact produced by the data pipeline. The model is consumed by the serving infrastructure; the serving infrastructure consumes the data pipeline. The data pipeline is the system; the model is a configuration of the system.
The model is versioned and evaluated
Every model version has a training set, an evaluation set, a metric, a deployment, a monitoring period, a retirement. The model lifecycle is managed as code (MLOps is the operational discipline; ML platforms are the infrastructure).
- Training set. The data the model learns from.
- Evaluation set. The data the model is judged against.
- Metric. The score the model is optimised for.
- Deployment. The promotion to production.
- Monitoring period. The window for drift.
- Retirement. The decommissioning.
The model evaluation is the per-request trust algorithm
Every model decision is evaluated against the inputs, the context, the model's confidence, the business rule. The model decision is a signal, not a decision. The signal is combined with other signals in the per-request authorisation model.
- Inputs. The features in the request.
- Context. The state of the system.
- Confidence. The model's certainty.
- Business rule. The policy that may override.
The monitoring is the operating model
The model is monitored for technical metrics (latency, throughput, error rate) and for business metrics (the metric the model is supposed to affect). The monitoring is in the same place as the data pipeline monitoring. The model is retired when the business metric stops improving.
§ 03 ·
State of the practice
3
MLOps maturity levels (Manual, CI/CD, Full MLOps) — Microsoft / Azure framework
Microsoft Azure MLOps maturity model
~80%
Of ML project effort is data wrangling, not model development
Industry observation (Anaconda State of Data Science, others)
1
Right answer to 'which ML platform' — the one the team will actually use
§ 04 ·
Questions we get asked
The metric that the model is supposed to affect. The technical metrics (accuracy, precision, recall) are useful for model development, but the business metric is the one that matters in production. The model evaluation is the per-request trust algorithm: every decision is evaluated against the inputs, the context, the model's confidence, and the business rule. The model decision is a signal, not a decision.
With monitoring and retraining. The model is monitored for performance against the business metric; the performance is compared to the baseline; the drift triggers a retraining. The retraining is part of the data pipeline. The model is retired when the retraining stops improving the business metric.
With the same per-request discipline as any other decision. The LLM is a component in the system; the system evaluates the LLM's output against the input, the context, the business rule. The LLM is not the system; the system is the system. The right answer is to treat the LLM as a model that has specific failure modes (hallucination, prompt injection, data leakage) and to design the system to address them. The threat model is the input; the controls are the system.
§ 05 ·
Patterns we design our practice to avoid
The model is an artifact produced by the data pipeline. The data pipeline is the system. A team that focuses on the model and ignores the data pipeline is a team that has the wrong object of attention.
The model is monitored for the metric the model is supposed to affect. The technical metrics (latency, throughput, error rate) are necessary but not sufficient. The business metric is the one that matters. A model that is fast and accurate and on stale data is a model that is making wrong decisions fast.
The model evaluation is on the live data. The test set is for development; the live data is for production. A model that is accurate on the test set and inaccurate on the live data is a model that has drifted.
The LLM is a component. The system is the system. The right answer is to treat the LLM as a model that has specific failure modes and to design the system to address them. A team that builds the system around the LLM is a team that has the wrong architecture.
§ 06 ·
Evidence & references
Public frameworks and writing that inform our practice.
Designing Machine Learning Systems
Huyen, O'Reilly, 2022
The most useful recent book on ML systems. The treatment of the data pipeline, the model lifecycle, and the monitoring is the most rigorous available. The book is not about MLOps specifically, but it is the foundation of the discipline.
Machine Learning Engineering
Burkov, Andriy Burkov, 2020
The book that organised the ML engineering discipline. The treatment of the project lifecycle, the model evaluation, and the deployment is the most useful single reference.
MLOps: Continuous delivery and automation pipelines in machine learning
Treveil et al., Packt, 2020
The most useful book specifically on MLOps. The treatment of the data pipeline, the model serving, and the monitoring is the most practical available.
Sculley et al., Google, NIPS 2015
The paper that framed the MLOps conversation. The treatment of the technical debt that accumulates in ML systems (data dependencies, model decay, feedback loops) is the foundation of the discipline.
Building an MLOps practice?
If you are weighing an MLOps initiative, evaluating a model serving platform, or trying to make an existing ML system more than an experiment, we are useful at the boundary between the data pipeline and the model. A short conversation is the right next step.
Learn more