Apache Airflow - Lineage
This article outlines the lineage coverage, configuration requirements, supported components, transformation support, supported scenarios, known limitations, and current functional status for lineage extraction in Apache Airflow.
The Airflow connector provides automated lineage extraction for Apache Airflow DAGs and SQL-based ETL tasks. It enables end-to-end visibility into orchestrated data flows by supporting table-level and column-level lineage across SQL pipelines spanning multiple database platforms.
Lineage Configuration Requirements
Successful lineage extraction depends on proper Airflow configuration, SQL metadata availability, and valid database connections. The following requirements must be satisfied to ensure accurate source-to-target lineage generation.
Configuration Requirements Table
SQL Dialects
SQL dialects must be defined in CrawlerOption.supportingDialects.
Airflow Connection
Airflow connection configuration is required through connectionInfoId.
Task SQL
Task queries must be available in JSON format with embedded SQL.
Database Connections
Database or catalog connections must exist in OvalEdge for schema and table resolution.
Metadata Availability
SourceCode and Dataset metadata must be available for lineage processing.
Missing SQL metadata, unavailable task definitions, unsupported SQL dialects, or unresolved database connections may result in incomplete lineage generation.
Lineage Components
DAGs
✅
SQL-based Tasks
✅
Static SQL Queries
✅
Multi-dialect SQL Parsing
✅
Table-level Lineage
✅
Column-level Lineage
✅
Views
✅
PythonOperator
⚠️
BashOperator
⚠️
BigQuery SQL
⚠️
Oracle SQL
⚠️
Amazon Redshift SQL
⚠️
Dynamic SQL / Variables
❌
Airflow Variables / Jinja Templates
❌
File-based ETL (CSV, Parquet, JSON)
❌
SparkSubmitOperator
❌
HttpOperator / API Lineage
❌
Kafka / Streaming Lineage
❌
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
Column Creation Support
This section describes lineage support for SQL-based column creation and transformation scenarios.
Column Creation from Lineage
Direct Column Mapping
✅
Column Aliases
✅
Expressions / Functions
✅
Aggregations
✅
CASE Statements
✅
Window Functions
✅
Join-based Columns
✅
Derived Columns
✅
Subqueries
✅
UNION / INTERSECT
✅
Dynamic Columns from Variables
❌
Python-generated Columns
❌
Column creation support applies only to SQL-based transformations extracted from Airflow task definitions.
Supported Use Cases
The connector supports lineage extraction across standard SQL-based Airflow orchestration workflows.
Supported Lineage Scenarios
DAG → Task → Table Lineage
Lineage extraction across DAGs, tasks, and database tables
SQL-based ETL Workflows
Lineage for SQL-driven ETL processing
Table-to-Table Lineage
Source-to-target table relationships
Column-level SQL Lineage
Column-level lineage generated from SQL statements
Join Operations
Lineage across SQL joins
Column Aliases
Resolution of aliased columns
Aggregations
Lineage for aggregation functions
CASE Statements
Lineage across conditional SQL expressions
Common Table Expressions (CTEs)
Lineage through CTE-based queries
Subqueries
Lineage for nested SQL queries
UNION / INTERSECT Operations
Lineage across SQL set operations
Multi-schema Lineage
Lineage across multiple schemas
Cross-database Lineage
Lineage across supported databases
Hive Workflows
SQL lineage for Hive tasks
Snowflake Workflows
SQL lineage for Snowflake tasks
SQL Server Workflows
SQL lineage for SQL Server tasks
MySQL Workflows
SQL lineage for MySQL tasks
View-based Lineage
Lineage across SQL views
Lineage extraction is based on SQL statements available within Airflow task definitions and the metadata available through configured database connections.
Partial or Limited Coverage
Certain scenarios provide partial lineage coverage due to SQL complexity, runtime behavior, or metadata limitations.
Scenarios
Dynamic SQL
Runtime variables and parameterized SQL are not resolved.
Use static SQL where possible.
PythonOperator
Supported only when SQL is embedded within the task JSON.
Embed SQL directly in the task definition.
BashOperator
Shell scripts are not parsed for lineage extraction.
Execute SQL through supported SQL-based tasks.
BigQuery, Oracle, and Amazon Redshift Operators
Require explicit SQL dialect configuration.
Configure the appropriate SQL dialect in CrawlerOption.supportingDialects.
Stored Procedures
Procedure calls are detected, but internal procedure logic is not parsed.
Validate procedure-level lineage separately where required.
TaskGroups / SubDAGs
Hierarchical workflow handling is limited.
Validate lineage for complex workflow hierarchies.
Jinja Templates
Template rendering is not supported during lineage extraction.
Use resolved SQL definitions where available.
Dataset Lineage Status
Lineage may remain incomplete when SQL extraction fails.
Verify SQL availability and metadata accessibility.
Runtime-generated SQL, template rendering, and unavailable SQL metadata may reduce lineage completeness and column-level accuracy.
Unsupported Scenarios
The connector does not support lineage extraction for non-SQL processing or runtime-generated execution patterns.
Unsupported Lineage
Python, Pandas, and PySpark Transformations
Non-SQL transformation logic is not parsed.
Document lineage manually or use supported SQL workflows.
SparkSubmitOperator
Spark job lineage is not extracted.
Capture lineage using supported Spark integrations where applicable.
File-based ETL
CSV, Parquet, and JSON processing is not analyzed.
Use supported database-based ingestion workflows.
API-based Processing
API execution lineage is not supported.
Document API lineage separately.
Kafka and Streaming Workflows
Streaming lineage is not available.
Capture lineage using supported streaming integrations where applicable.
Airflow Variables
Variable substitution is not resolved during parsing.
Use static SQL definitions where possible.
XCom Data Movement
Data exchanged through XCom is not tracked.
Document dependencies manually where required.
External Shell Scripts
External script execution is not parsed.
Execute SQL through supported Airflow SQL tasks.
Custom Operators Without SQL Exposure
Operators without accessible SQL cannot be analyzed.
Use SQL-based operators or document lineage manually.
Unsupported scenarios do not generate lineage and may appear as disconnected nodes within lineage visualization.
Current Functional Status
This section summarizes the current lineage capabilities available for the Apache Airflow connector.
Overall Coverage
Strong coverage for SQL-based Airflow workflows
Lineage Depth
Table-level and column-level SQL lineage
Supported Inputs
DAGs, SQL-based tasks, static SQL queries, views, and supported SQL dialects
Functional Scope
SQL lineage extraction across ETL orchestration workflows
Limitation Areas
Dynamic SQL, runtime variables, templates, Python transformations, file-based processing, and non-SQL operators
Resulting Output
Reliable lineage for standard SQL-based Airflow workflows with partial coverage for advanced orchestration scenarios
The Apache Airflow connector provides strong lineage coverage for SQL-based DAGs and ETL workflows, including table-level and column-level lineage across supported SQL dialects. Advanced scenarios involving dynamic SQL, runtime variables, Python transformations, templates, and non-SQL processing have partial or unsupported coverage and may require manual validation.
Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA
Last updated
Was this helpful?