Data Lineage for AI: Tracing Every Prediction to Its Source

Paweł Szczepanik
Paweł Szczepanik
November 20, 2026
•
8 min read
Loading the Elevenlabs Text to Speech AudioNative Player...

Data lineage for AI is the chain of records that links one prediction to the model version that made it, the training snapshot behind that version, the jobs that built the snapshot and the source columns they read. Analytics lineage ends at a table or a dashboard; AI lineage has to cross the model registry and the serving log. Most data lineage tools capture the table and column links on their own. The chain usually breaks at the two links nobody captures by default: the exact training snapshot and the logged prediction. Article 12 of the EU AI Act (Regulation (EU) 2024/1689) ties logging to "a level of traceability of the functioning of a high-risk AI system," and for high-risk systems in Annex III it applies from December 2, 2027.

What Data Lineage for AI Has to Cover

In AI systems, data lineage is the record of how data moved and changed from source systems to a model's output, kept at a grain fine enough to answer for a single prediction. It overlaps with provenance, which W3C's PROV standard defines as "information about entities, activities, and people involved in producing a piece of data or thing." Provenance says where something came from. Lineage is the sequence of steps between that origin and the result. A regulator asks about the first, an incident review about the second.

Compared with a BI chain, a model adds two objects a data catalog does not see: the snapshot that training read and the log of what the served model answered. Broken into records, data lineage for one prediction has five links:

LinkQuestion it answersRecord that proves itCaptured automatically?
1. Source columnWhich raw fields fed the pipeline?Column-level edge in the catalogOften, inside one platform
2. Job runWhich run of which code produced the table?Run event with job name, inputs and outputsOften, if the orchestrator emits events
3. Training snapshotWhich exact rows did training read?Dataset version or digest logged with the training runRarely, unless training code logs it
4. Model versionWhich version was trained on that snapshot?Registry entry linked to the runPartly, through the model registry
5. PredictionWhich version answered, when, with what input?Inference log row with request ID and timestampOnly if logging is switched on

Audits ask about links three and five. Tools capture one and two.

Where lineage sits next to catalogs, access policy and audit is covered in our guide to governing data across AI programs; this article stays on one chain: the records that connect a single prediction to its sources, and the places that chain breaks.

What the EU AI Act and NIST Actually Ask For

For high-risk systems, the AI Act describes this chain with three other words. Article 10(2)(b) covers "data collection processes and the origin of data." Annex IV, the technical documentation annex, asks in point 2(d) for "information about their provenance, scope and main characteristics; how the data was obtained and selected." Article 12(2) asks for logs that give traceability "appropriate to the intended purpose of the system."

The articles that set these duties never use the word lineage. They ask for origin, provenance and traceability, which is data lineage described by what it has to prove.

They also start two clocks. Under Article 18(1), the provider keeps technical documentation "for a period ending 10 years after the high-risk AI system has been placed on the market or put into service." Article 19(1) for providers and Article 26(6) for deployers require automatically generated logs to be kept for at least six months, unless other law provides otherwise. Training set provenance lands in a document that must last a decade, so data lineage records belong in it, not only in a tool. Under Article 113, as amended by Regulation (EU) 2026/1744, these articles apply to high-risk systems from December 2, 2027 for Annex III and August 2, 2028 for Annex I.

NIST's voluntary AI RMF 1.0 (NIST AI 100-1, January 2023) puts it more softly: "Maintaining the provenance of training data and supporting attribution of the AI system’s decisions to subsets of training data can assist with both transparency and accountability."

Which AI Act obligations apply to a given system is a classification question for another day; the Act appears here only where it names the origin of data, its provenance, traceability and how long records must be kept.

Links One and Two: Column-Level Data Lineage From Job Runs

The first two links come from job runs. OpenLineage "defines a generic model of dataset, job, and run entities uniquely identified using consistent naming strategies," and its column facet records "which input columns are used to produce which output columns and in what way," with two types. DIRECT means the output value comes from the input; INDIRECT means the input shapes the output without supplying its value, like a filter. For a model, indirect dependencies often matter more: a join filter decides which customers enter training at all.

For systems outside Google Cloud, Google's Knowledge Catalog documentation says: "We recommend importing events formatted according to the OpenLineage standard." Databricks says Unity Catalog records these links "automatically for queries run on Databricks, down to the column level."

The same documentation shows where data lineage breaks. Databricks (docs updated November 2026): "Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns." Column links appear only when source and target are named as tables, not file paths. UDFs get table-level links only. The system tables page adds that they "represent a subset of all read/write events."

Any data lineage captured from queries shares this edge: code that bypasses the engine leaves no trace. Links one and two stay cheap when every job emits events in one format and names tables instead of paths.

Link Three: Pin the Training Snapshot, Not Just the Table

Table-level links say training read the customers table. An audit asks which rows, in which version, of a table that has changed many times since.

MLflow's dataset tracking logs two fields on the training run: a "Digest: Unique hash/fingerprint for dataset identification (automatically computed)" and a "Source" pointing to the original data location. AWS is candid about the limit in SageMaker ML Lineage Tracking: you can reproduce the model "as long as you have access to the exact dataset that was used." Training data lineage stores a pointer, not the data. If nobody keeps the snapshot, the pointer leads nowhere.

External data needs one more check. The Data Provenance Initiative (Longpre et al., 2023) audited more than 1,800 text datasets and found that widely used dataset hosting sites frequently miscategorize licenses, with license omission of 70%+ and error rates of 50%+. For downloaded data, the source link has to be verified, not copied from a hosting page. Datasheets for Datasets (Gebru et al., 2018) offers a format for recording it.

In practice, link three means versioned training and test datasets tied to each model run, stored where the model documentation can point to them.

Rebuilding features as of a past date is a feature store problem; lineage asks a narrower question: which exact snapshot did this model version read.

Links Four and Five: From Model Version to the Logged Prediction

Link four is the most automated AI-specific link, since registries record which run produced which version. In Unity Catalog's graph, "Nodes can represent tables and views, ML model versions, external assets, and file paths."

Link five exists only if someone switched it on. Databricks inference tables "log requests and responses from your model services to Unity Catalog Delta tables," with request_id, event_time, request and response in each row, and "Databricks charges for the requests and responses that inference tables log." Because logging costs money, decide what to log and for how long before an audit, not during one.

Retention is where model and data lineage meet the calendar. Databricks system tables "retain a rolling 395-day window of data by default," while Catalog Explorer and the API keep records captured after September 1, 2024 indefinitely. That covers six months of logs but not 10 years of documentation. Our order: export the snapshot digest, training run ID and model version into each model's release documentation. The tool's graph is an index, not an archive.

The same inference log feeds drift monitoring, but here it matters as evidence: the one record that ties an output to a model version and a timestamp.

Lineage for RAG and Generative AI

Retrieval-augmented generation keeps the five links but changes two. Link three becomes the version of the index and of the documents its embeddings came from. Link five becomes the answer plus the IDs of the chunks retrieved for it. Without these two records, nobody can tell whether a wrong answer came from the model, a document or a stale index. Data lineage for a RAG system has to log both, per answer.

For a vendor's foundation model, the source link is a document. Through Article 53(1)(b), Annex XII point 2(c) requires providers of general-purpose AI models to give integrators information on training data, "including the type and provenance of data and curation methodologies," a duty in force since August 2, 2025. Keep that document's version next to your own system's version.

Chunk IDs and index versions are rarely logged by default. Setting up lineage for retrieval sources and prompts in generative AI follows the same five-link test as any other model.

How We Test a Lineage Chain: One Prediction, Five Hops

Our test starts from the output and walks back:

  • Pick one production prediction from a month ago.
  • From the inference log, read the model version and timestamp.
  • From the registry, open the training run behind that version.
  • From the run, follow the dataset digest to the snapshot.
  • From the snapshot, trace the jobs and source columns that built it.

At each hop we note whether a record existed or someone had to be asked. Every hop that needed a person is a link to fix. We fix from the end, prediction log first and snapshot second, because a missing prediction log cannot be backfilled, while column-level data lineage can be re-captured by rerunning jobs. To find where your own chain breaks, pick one production prediction and we will walk the chain with you.

FAQ

What is the difference between data lineage and data provenance?

Data lineage for AI follows data through every step from source to output; provenance describes where a piece of data came from and who produced it. An AI audit needs both.

Does the EU AI Act require lineage records?

Not by that name. For high-risk systems it requires governance of the origin of training data (Article 10(2)(b)), provenance in the technical documentation (Annex IV) and logs that provide traceability (Article 12(2)). These apply from December 2, 2027 for Annex III systems.

Is table-level lineage enough for machine learning?

Rarely. A model depends on specific columns and on the filters that pick training rows, which OpenLineage records as direct and indirect dependencies. Table-level links also cannot say which version of a table a run read; that needs a dataset digest logged with the run.

How long should AI lineage records be kept?

As long as the documentation they support. High-risk providers keep technical documentation for 10 years and automatically generated logs for at least six months, unless other law sets a different period. Tool defaults are often shorter: Databricks system tables keep a rolling 395 days.

Share this post
Data Engineering
Paweł Szczepanik
MORE POSTS BY THIS AUTHOR
Paweł Szczepanik

Curious how we can support your business?

TALK TO US

More insights

More news

No items found.