A feature store machine learning teams keep hearing about is a contract, not a database: one feature definition produces the same value for the row a model trains on and the row it scores, including the value as it stood on the label date. Google's Rules of Machine Learning, written by Martin Zinkevich, give training-serving skew three causes. A feature store removes the first, a mismatch in how the training and serving pipelines handle data, and it makes old training sets reproducible. Uber's 2017 Michelangelo post reported "approximately 10,000 features in Feature Store," a scale at which sharing alone justifies the contract. One batch model, owned by one team and scored from the table it trained on, gets the same contract for free.
What a Feature Store Machine Learning Teams Build Actually Does
A feature store is a registry of feature definitions plus two stores: an offline store that keeps the full history for training and batch scoring, and an online store that keeps the latest value per entity for low-latency prediction. Both are computed by the same definition, and training reads are taken as of each label's timestamp.
The feature store machine learning platforms bundle today follows that shape. Databricks calls its product "a central registry for the features used in your AI and ML models." Amazon SageMaker's developer guide says "The online store retains only the latest records for your features," while "The offline store keeps all records for your features as a historical database."
The Feast documentation is more useful for what a store is not: "not a general purpose data pipelining system," "not a replacement for your data warehouse," "not a database." Your pipelines still compute the values.
This guide settles whether a specific team needs a separate system for features, and what it takes on if the answer is yes. Where a feature store sits next to the lakehouse, the vector index and the model registry is covered in how to choose a data platform for AI workloads.
Which Cause of Training-Serving Skew a Feature Store Removes
Zinkevich lists "A discrepancy between how you handle data in the training and serving pipelines," "A change in the data between when you train and when you serve," and "A feedback loop between your model and your algorithm." A feature store turns Rule #32, "Re-use code between your training pipeline and your serving pipeline whenever possible," into a constraint: one definition, called by both paths.
The first stores were built for exactly that. Uber's Michelangelo used a DSL so that "the same expressions are applied at training time and at prediction time." LinkedIn's post on open sourcing Feathr (2022) set the goal "to ensure features are prepared in the same way for the training and inferencing contexts."
The other two causes are not for sale. Changing data is a monitoring job: Google built a separate data validation system for TFX (SysML 2019) to check several petabytes of production data per day, training/serving skew included, and Rule #37 reads "Measure Training/Serving Skew." Feedback loops are a design question. When Databricks' documentation claims its approach "eliminates training/serving skew," read that as the first cause only.
Why Last Year's Training Set Is Hard to Rebuild
Take a March label and the feature "transactions in the last month," computed today. Joined to that label, it leaks the future into training, and the model looks better offline than in production. Rule #31 warns that "if you join data from a table at training and serving time, the data in the table may change."
The second thing a feature store machine learning teams rely on buys them is a training set as of a past date. Feast can "reproduce the state of features at a specific point in the past." In Databricks, "any table with a TIMESERIES primary key is a time series feature table." Hopsworks builds training data "with no data leakage."
Without a store you have two routes. Log-and-wait (Rule #29) saves the features used at serving time and trains on that log; the README of the Chronon project, published under Airbnb's GitHub organization, names the cost: "It might take a long to accumulate enough data to train the model." A backfill is faster, and Chronon's "Backfills are automatically point-in-time correct, which avoids label leakage," as are Tecton's. Hand-written backfills are where leakage slips in. Inside a warehouse, a slowly changing dimension does this for attributes the warehouse holds; a feature store does it for features computed outside it and served online.
Four Signals You Need One, From the Teams That Reported Them
Four signals separate the feature store machine learning organizations needed from the one they were sold, and each was reported by a team in the first person.
Shared features come first. Uber had approximately 10,000 features in its store by 2017 and saw "substantial value in enabling teams to share features between their own projects"; LinkedIn runs "hundreds of ML models" on "thousands of features." Online serving is second: Uber kept features for online models "precomputed and stored in Cassandra," updated daily, and SageMaker's online store targets "low millisecond latency reads", a description of the serving mode. Historical rebuilds are third, and parallel pipelines fourth: at LinkedIn, "The cost of building and maintaining feature pipelines was borne redundantly across many teams," and Feathr cut the time to add new features "from weeks to days." A 2023 preprint from Microsoft's feature store team says that without a store, teams "would maintain the above process independently."
| Signal | What it needs | What covers it without a feature store | Who reported it |
|---|---|---|---|
| 1. The same features feed more than one model | One definition per feature, an owner, a place to find it | A shared SQL view, until the second team edits it | Uber; LinkedIn |
| 2. Predictions are served online from precomputed values | Latest value per entity, refreshed on a schedule the model tolerates | A key-value cache fed by a batch job, unlinked from the training definition | Uber; SageMaker documentation |
| 3. Training sets must be rebuilt as of past dates | Timestamped history and a point-in-time join | A Type 2 table in the warehouse, or log-and-wait, which costs calendar time | Feast, Databricks and Hopsworks documentation; Chronon |
| 4. Several teams maintain parallel pipelines for the same features | A registry both teams write to, with tests on shared definitions | A naming convention and code review, until the teams stop sharing a repository | LinkedIn; Microsoft's 2023 preprint |
One signal alone is a reason to watch. The first two together are the common real case; the third suffices alone when labels arrive months late. With none, keep the keyed table.
When a Keyed Table Is Enough
Most of the feature store machine learning workloads need for one batch model already exists as a keyed table, and the platforms say so. Databricks: "In Unity Catalog, any Delta table with a primary key constraint can serve as a feature table." Google's documentation for Vertex AI Feature Store, now Feature Store on Gemini Enterprise Agent Platform: "the BigQuery tables or views containing the feature data collectively form the offline store." SageMaker's offline store is "an append-only store" of Parquet files in your own S3 bucket. What you actually add is the online store and the registry.
We would not build one when:
- One batch model and one team score from the same table the model trained on.
- Features come from one source and are computed with plain SQL in one place.
- Nothing is in production yet. Get the model working, then add the contract.
- The real problem is data quality on the inputs, which a registry will not fix.
Sculley and coauthors (2015) predate the term but name both failure modes. Without a shared contract, "pipeline jungles often appear in data preparation." With a store nobody needed, you add glue to a system that "might end up being (at most) 5% machine learning code and (at least) 95% glue code."
A Feature Store Is a Production System, and Kpler Shows the Cost
Kpler, a market intelligence and data analytics company, already ran a feature store when DS Stream joined. The symptoms were "long data processing times, database connection issues, and lack of observability in model training." The work added monitoring to the ingestion process, fixed connection pool exhaustion with a read replica set up alongside Kpler's DevOps team, and introduced unit tests, failure handling and a new framework for model access, on AWS, EKS and Argo CD. The case study describes how Kpler's feature store ingestion went from four hours to ten minutes.
The lesson for a yes decision is a list of duties. Someone owns the ingestion SLA: how long a refresh takes and whether models wait for it. The store is a database client like any other, so connection pools and replicas become yours. Training needs observability, and model runs need logs that survive. None of this is in a feature store's documentation, and all of it was in Kpler's ticket.
How to Decide in One Afternoon, and How We Approach It
- List the features of every current and planned model; mark the repeats.
- Note whether each model scores in batch or online, and where its values come from at prediction time.
- Try to rebuild last year's training set from what you have. If you cannot, that is the third signal.
- With no signals, keep a table with a primary key and a timestamp in your existing catalog, and make scoring read from the table training used.
- With signals, start from what your platform bundles and run ingestion as a service with an SLA from day one.
The feature store machine learning teams regret is the one built before the second model.
A feature store enters our projects when the feature review shows signals; that is how the MLOps practice we build around the models you already run works. When the signals are there, we deliver a feature store built on the platform you already have, with ingestion treated as a service from day one. When a store already exists and has become the bottleneck, we start with ingestion and keep the tool. For a second opinion, bring us the feature list of one production model and the one you plan next.
FAQ
Do you need a feature store for a single batch model?
The feature store machine learning teams run for a single batch model is usually overhead. If it trains and scores from one keyed table, training and serving already share one definition. A store starts paying off when a second model reuses the features or predictions move online.
What is the difference between a feature store and a data warehouse?
A warehouse stores and queries data. A feature store adds a registry of definitions, point-in-time retrieval for training and an online path for low-latency reads. Databricks treats a keyed Delta table as a feature table and Google treats BigQuery tables as the offline store, so in practice the warehouse is the offline store.
How does a feature store prevent training-serving skew?
It removes the first cause in Google's Rules of Machine Learning, the discrepancy between how training and serving pipelines handle data, because both paths compute from one feature definition. The other two causes, changing data and feedback loops, need monitoring and system design.
Should you use Feast or the feature store built into your cloud platform?
Start with what your platform bundles. Databricks, SageMaker and Vertex AI each offer an offline store over tables you have and an online store you can switch on. Feast fits when one definition layer must span several platforms.
