OpenLineage

Free plan APISelf-hosted

What OpenLineage is

OpenLineage is a free, open-source framework for collecting and analyzing lineage metadata about datasets, jobs, and runs. It defines an extensible open standard for these entities, with user-defined facets that add organization-specific metadata. The specification covers runtime events such as job-run states and design-time information about jobs and datasets, supporting root-cause investigation and impact analysis when data changes. OpenLineage is available through an API and self-hosted deployments. It is a specification accompanied by integrations, examples, and tools, not a metadata server; the project identifies Marquez as a server that can receive and analyze its events. Deployments may use one consumer or several. Documented integrations include Apache Airflow, Flink, Hive, Spark, dbt, Feast, Great Expectations, SQL, and Trino. Python and Java clients emit events to HTTP, Kafka, and other consumers, while the Python client also documents file and console transports and defaults to console output without configuration. A SQL parser is offered as Rust, Java, or Python libraries. Spark can collect metadata from filesystems, JDBC backends, and data warehouses. dbt supports post-run artifact parsing or real-time structured JSON logs. OpenLineage is an LF AI & Data Foundation project with Slack community support and technical steering meetings.

Who it suits

OpenLineage suits organizations that depend on data operations and teams building data-processing products or technology. It is aimed at users who need automated discovery, column-level lineage, pipeline lineage, impact analysis, and integrations across multiple processing systems.

What it does well

Free and open-source open standard
Tracks datasets, jobs, and runs
User-defined facets extend the lineage model
Python and Java event clients
Documented integrations across major data tools
Supports automated discovery and impact analysis

What to know first

It is not a metadata server
Integration capability matrix is incomplete
Legacy Airflow support before version 2.7 is unmaintained
Not every mechanism gathers every metadata facet

Limits

The integration matrix is explicitly incomplete, and individual mechanisms may collect only some metadata facets. The legacy Airflow integration for versions earlier than 2.7 receives no bug fixes, security updates, or new features.

Getting started

Open the OpenLineage website
Choose an API or self-hosted deployment
Select a documented integration or client library
Configure an event consumer such as HTTP, Kafka, file, or console
Connect a server such as Marquez to receive and analyze events

Verdict

Choose OpenLineage when you need an open, extensible format for lineage events across data tools. Look elsewhere if you want a ready-made metadata server; the project points to Marquez for receiving and analyzing events.

Questions

Is OpenLineage free?

Yes. It is free and open source.

What platforms does OpenLineage support?

It is available as an API and for self-hosted deployment.

Is OpenLineage a metadata server?

No. It is a specification with integrations, examples, and tools; Marquez is identified as a server that can receive and analyze its events.

Which integrations are documented?

Documented integrations include Apache Airflow, Flink, Hive, Spark, dbt, Feast, Great Expectations, SQL, and Trino.

Can I create custom integrations?

Python and Java clients can emit events to HTTP, Kafka, and other consumers. A SQL parser is available as Rust, Java, or Python libraries.

Facts from OpenLineage's own pages

Airflow version support: The Airflow integration has moved to the official provider; the legacy integration for Airflow versions earlier than 2.7 is no longer maintained and receives no bug fixes, security updates, or new features. · openlineage.io, 11 Oct 2026
Backend requirement: OpenLineage is a specification with integrations, examples, and tools, rather than a metadata server; the FAQ points to Marquez as a server that can receive and analyze its events. · openlineage.io, 11 Oct 2026
Capability limit: The integrations capability matrix is explicitly marked incomplete, and not every mechanism gathers every metadata facet. · openlineage.io, 11 Oct 2026
Client libraries: OpenLineage provides Python and Java clients for creating and emitting lineage events. · openlineage.io, 11 Oct 2026
Community support: The project directs users to its Slack community for help and provides open technical steering committee meetings to discuss project updates. · openlineage.io, 11 Oct 2026
Custom integrations: Python and Java clients can emit events to HTTP, Kafka, and other consumers, and the project provides a SQL parser available as Rust, Java, or Python libraries. · openlineage.io, 11 Oct 2026
dbt modes: The dbt integration can parse generated artifacts after a run or consume structured JSON logs in real time; artifact parsing waits for the dbt command to finish. · openlineage.io, 11 Oct 2026
Deployment: The project describes simple deployments with single consumers and complex deployments with multiple consumers. · openlineage.io, 11 Oct 2026
Event types: The specification includes runtime job run state events and design-time job and dataset metadata events. · openlineage.io, 11 Oct 2026
Extensible model: Its model represents datasets, jobs, and runs, and can be extended with user-defined facets. · openlineage.io, 11 Oct 2026
Integration limits: The integrations capability matrix is explicitly incomplete, and the compatibility matrix lists minimum versions for some supported platforms. · openlineage.io, 11 Oct 2026
Integrations: Documented integrations include Apache Airflow, Flink, Hive, Spark, dbt, Feast, Great Expectations, SQL, and Trino. · openlineage.io, 11 Oct 2026
Intended users: The project describes its audience as organizations that rely on data in their operations and organizations building products or technology to manage data and its processing. · github.com, 11 Oct 2026
Not a metadata server: OpenLineage is a specification with integrations, examples, and tools; the FAQ points to Marquez for receiving and analyzing events. · openlineage.io, 11 Oct 2026
Open standard: The project describes its specification as an open standard that enables systems to interoperate with lineage metadata. · openlineage.io, 11 Oct 2026
Project governance: OpenLineage is an LF AI & Data Foundation project, and its site welcomes contributions and community feedback. · openlineage.io, 11 Oct 2026
Purpose: OpenLineage is an open framework for collecting and analyzing data lineage metadata about datasets, jobs, and runs. · openlineage.io, 11 Oct 2026
Security: The legacy Airflow integration is explicitly described as no longer receiving security updates. · openlineage.io, 11 Oct 2026
Security and compliance: The opened official pages do not state a security certification or compliance claim. · openlineage.io, 11 Oct 2026
Spark metadata: The Spark integration uses SparkListener events to collect job and dataset metadata from sources including filesystems, JDBC backends, and data warehouses. · openlineage.io, 11 Oct 2026
Standard: Its extensible open standard defines a generic model for dataset, job, and run entities, with user-defined facets for additional metadata. · openlineage.io, 11 Oct 2026
Transport options: The clients can emit events to HTTP, Kafka, and other consumers. · openlineage.io, 11 Oct 2026
What it tracks: It tracks metadata about datasets, jobs, and runs to help identify root causes and understand the impact of changes. · openlineage.io, 11 Oct 2026

Where OpenLineage stands on Freedom251

Data Lineage Tools#13 of 43