Feature Stores and Lineage: Reproducible, Auditable Machine Learning

The question every ML team eventually faces

Sooner or later someone asks a simple question: why did the model make this decision? Answering it means knowing which model version ran, which features it saw, how those features were computed and which source data they came from. In many organisations, nobody can say.

The problem is rarely the model. It is the data around it: features computed one way for training and another way in production, datasets overwritten in place, pipelines that fail silently. Feature stores, lineage and quality gates are the tools that fix it.

1. One definition per feature

The most common source of silent errors in production ML is training–serving skew: a feature calculated slightly differently when the model learns and when it predicts. A feature store removes it by making each feature a single, shared definition.

• Define each feature once, with an owner, a description and the logic that computes it.
• Serve the same definition to training jobs and to real-time predictions.
• Reuse features across models instead of rebuilding them, so improvements benefit every model at once.

2. Version your datasets

A result you cannot rebuild is a result you cannot trust. Versioning datasets, not just code, is what makes machine learning reproducible.

• Snapshot or version every dataset used for training and validation, and never overwrite one in place.
• Record which dataset version, code version and parameters produced each model.
• Keep enough history to rebuild any model still in production.

3. Trace lineage from source to prediction

Lineage links everything together: source systems, transformations, features, datasets and models. With it, a question about one prediction becomes a lookup rather than an investigation.

• Capture lineage automatically from your pipelines rather than documenting it by hand.
• Make it possible to go from a prediction to the model version, features and source data behind it.
• Use lineage in the other direction too: when a source changes, see every feature and model it affects.

4. Put quality gates on every pipeline

Bad data is cheaper to stop at the door than to find in a model's behaviour weeks later. Quality gates make that the default.

• Check schema, freshness, completeness and value ranges at each pipeline stage.
• Stop the pipeline when a check fails, instead of passing bad data downstream.
• Write validation standards down, so every new dataset meets the same bar.

5. Govern datasets like the assets they are

Training, validation and inference datasets carry legal and business risk as well as value. Governance makes that risk visible and manageable.

• Give each dataset an owner, a purpose and a retention rule.
• Control access, and minimise personal data to what the model genuinely needs.
• Keep the documentation and lineage auditors and risk teams ask for, designed with GDPR and the EU AI Act in mind.

Questions you should be able to answer

A simple test of whether your ML data is under control: can you answer each of these questions in minutes, not days?

• Which model version produced this prediction, and which features did it use?
• Are these features computed the same way in training and in production?
• Which dataset version trained the model currently in production, and can we rebuild it?
• If this source system changes, which features and models are affected?
• Who owns this dataset, why do we hold it, and how long will we keep it?

Start where the pain is

You do not need a full data platform to begin. Start with the model that is hardest to explain or most often wrong in production: version its training data, move its features into a shared definition, and add quality checks to its pipeline.

Each step makes one model reproducible and auditable. Repeated across your models, it gives you something rarer: machine learning you can explain to anyone who asks.