Data Lineage

Tracking Data Origins, Transformations & Dependencies in AI/ML Systems

What is Data Lineage?

Data lineage is the complete lifecycle of data—from its origin through every transformation, aggregation, and destination. In AI/ML systems, it answers critical questions: "Where did this data come from?", "How was it transformed?", and "What systems depend on it?"

For LLM applications, lineage extends to prompt evolution, training data provenance, and RAG document sources—essential for debugging, compliance, and maintaining trust.

Origin

Data Source

Transform

Processing

Depend

Downstream

Impact

Analysis

Why Data Lineage Matters for AI

Debugging & Root Cause Analysis

Critical

When an LLM produces incorrect output, lineage helps trace back to the root cause—was it bad training data, a faulty RAG document, or a broken preprocessing step?

Regulatory Compliance

GDPR/CCPA

Regulations like GDPR require knowing where personal data flows. Lineage enables "right to be forgotten" by identifying all derived data that must be deleted.

Impact Analysis

Change Mgmt

Before changing a data source or schema, understand what downstream models, pipelines, and dashboards will be affected. Prevents cascading failures.

Reproducibility

ML/AI

Recreate any historical model state by knowing exactly which data version, preprocessing code, and configuration was used at that point in time.

Data Lineage Flow

flowchart LR subgraph Sources["Data Sources"] S1[(Database)] S2[(API)] S3[(Files)] end subgraph Processing["Transformations"] T1[Clean] T2[Validate] T3[Enrich] T4[Embed] end subgraph Storage["Data Storage"] D1[(Vector DB)] D2[(Feature Store)] end subgraph Consumers["AI/ML Systems"] C1[RAG Pipeline] C2[LLM] C3[Dashboard] end S1 --> T1 S2 --> T1 S3 --> T1 T1 --> T2 T2 --> T3 T3 --> T4 T4 --> D1 T3 --> D2 D1 --> C1 D2 --> C1 C1 --> C2 D2 --> C3

Lineage captures the complete path from source to consumer, including all transformations.

Types of Data Lineage

Table-Level Lineage

Tracks relationships between tables and datasets. "Table A feeds into Table B."

Coarse-grained

Column-Level Lineage

Tracks individual column transformations. "Column X derived from columns A + B."

Fine-grained

Row-Level Lineage

Tracks individual records through transformations. Essential for GDPR.

Most detailed

LLM-Specific Lineage

LLM applications require tracking additional lineage dimensions:

Prompt Lineage

Track prompt versions, A/B tests, and which prompt produced which output.

RAG Document Lineage

Know which source documents contributed to each RAG response.

Training Data Lineage

Track datasets used for fine-tuning and their preprocessing steps.

Model Lineage

Connect model versions to their training data, hyperparameters, and code.

Tracking Lineage with OpenLineage

Python - OpenLineage Integration

from openlineage.client import OpenLineageClient
from openlineage.client.facet import (
    DataSourceDatasetFacet,
    SchemaDatasetFacet,
    SchemaField
)
from openlineage.client.run import (
    RunEvent, RunState, Run, Job,
    InputDataset, OutputDataset
)
import uuid
from datetime import datetime

# Initialize OpenLineage client
client = OpenLineageClient(url="http://localhost:5000")

# Define the job (transformation)
job = Job(namespace="rag-pipeline", name="document-embedder")

# Define input datasets
inputs = [
    InputDataset(
        namespace="raw-documents",
        name="confluence-docs",
        facets={
            "schema": SchemaDatasetFacet(
                fields=[
                    SchemaField(name="doc_id", type="STRING"),
                    SchemaField(name="content", type="TEXT"),
                    SchemaField(name="updated_at", type="TIMESTAMP")
                ]
            )
        }
    )
]

# Define output datasets
outputs = [
    OutputDataset(
        namespace="vector-store",
        name="pinecone-embeddings",
        facets={
            "schema": SchemaDatasetFacet(
                fields=[
                    SchemaField(name="vector_id", type="STRING"),
                    SchemaField(name="embedding", type="VECTOR(1536)"),
                    SchemaField(name="metadata", type="JSON")
                ]
            )
        }
    )
]

# Emit start event
run_id = str(uuid.uuid4())
client.emit(RunEvent(
    eventType=RunState.START,
    eventTime=datetime.now().isoformat(),
    run=Run(runId=run_id),
    job=job,
    inputs=inputs,
    outputs=[]
))

# ... perform embedding work ...

# Emit complete event
client.emit(RunEvent(
    eventType=RunState.COMPLETE,
    eventTime=datetime.now().isoformat(),
    run=Run(runId=run_id),
    job=job,
    inputs=inputs,
    outputs=outputs
))

Lineage Tools & Platforms

Tool	Type	Best For
OpenLineage	Open Standard	Interoperability
Marquez	Open Source	OpenLineage backend
Apache Atlas	Open Source	Hadoop ecosystem
DataHub	Open Source	Data catalog + lineage
dbt	Open Source	SQL transformations
Collibra	Enterprise	Enterprise governance

Best Practices

Automate Collection

Use instrumentation libraries (OpenLineage, dbt) to capture lineage automatically rather than manual documentation.

Tag Everything

Add metadata tags for PII, sensitivity level, data owner, and retention policies to every dataset.

Version Data

Combine lineage with data versioning (DVC, lakeFS) for complete reproducibility.

Visualize Regularly

Create lineage graphs that teams can explore. Visual lineage reveals hidden dependencies.

Test Your Knowledge

Score 8/10 or higher to pass