> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/release8.2/connectors/connector-repositories/etl-tool/apache-airflow/apache-airflow-lineage.md).

# Apache Airflow - Lineage

This article outlines the lineage coverage, configuration requirements, supported components, transformation support, supported scenarios, known limitations, and current functional status for lineage extraction in Apache Airflow.

The Airflow connector provides automated lineage extraction for Apache Airflow DAGs and SQL-based ETL tasks. It enables end-to-end visibility into orchestrated data flows by supporting table-level and column-level lineage across SQL pipelines spanning multiple database platforms.

## Lineage Configuration Requirements

Successful lineage extraction depends on proper Airflow configuration, SQL metadata availability, and valid database connections. The following requirements must be satisfied to ensure accurate source-to-target lineage generation.

### Configuration Requirements Table

<table><thead><tr><th width="212">Configuration</th><th>Required Detail</th></tr></thead><tbody><tr><td>SQL Dialects</td><td>SQL dialects must be defined in CrawlerOption.supportingDialects.</td></tr><tr><td>Airflow Connection</td><td>Airflow connection configuration is required through connectionInfoId.</td></tr><tr><td>Task SQL</td><td>Task queries must be available in JSON format with embedded SQL.</td></tr><tr><td>Database Connections</td><td>Database or catalog connections must exist in OvalEdge for schema and table resolution.</td></tr><tr><td>Metadata Availability</td><td>SourceCode and Dataset metadata must be available for lineage processing.</td></tr></tbody></table>

{% hint style="warning" %}
Missing SQL metadata, unavailable task definitions, unsupported SQL dialects, or unresolved database connections may result in incomplete lineage generation.
{% endhint %}

## Lineage Components

| Component                           | Supported |
| ----------------------------------- | :-------: |
| DAGs                                |     ✅     |
| SQL-based Tasks                     |     ✅     |
| Static SQL Queries                  |     ✅     |
| Multi-dialect SQL Parsing           |     ✅     |
| Table-level Lineage                 |     ✅     |
| Column-level Lineage                |     ✅     |
| Views                               |     ✅     |
| PythonOperator                      |     ⚠️    |
| BashOperator                        |     ⚠️    |
| BigQuery SQL                        |     ⚠️    |
| Oracle SQL                          |     ⚠️    |
| Amazon Redshift SQL                 |     ⚠️    |
| Dynamic SQL / Variables             |     ❌     |
| Airflow Variables / Jinja Templates |     ❌     |
| File-based ETL (CSV, Parquet, JSON) |     ❌     |
| SparkSubmitOperator                 |     ❌     |
| HttpOperator / API Lineage          |     ❌     |
| Kafka / Streaming Lineage           |     ❌     |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

## Column Creation Support

This section describes lineage support for SQL-based column creation and transformation scenarios.

### Column Creation from Lineage

| Transformation Type (SQL Queries Only) | Supported |
| -------------------------------------- | :-------: |
| Direct Column Mapping                  |     ✅     |
| Column Aliases                         |     ✅     |
| Expressions / Functions                |     ✅     |
| Aggregations                           |     ✅     |
| CASE Statements                        |     ✅     |
| Window Functions                       |     ✅     |
| Join-based Columns                     |     ✅     |
| Derived Columns                        |     ✅     |
| Subqueries                             |     ✅     |
| UNION / INTERSECT                      |     ✅     |
| Dynamic Columns from Variables         |     ❌     |
| Python-generated Columns               |     ❌     |

{% hint style="info" %}
Column creation support applies only to SQL-based transformations extracted from Airflow task definitions.
{% endhint %}

## Supported Use Cases

The connector supports lineage extraction across standard SQL-based Airflow orchestration workflows.

### Supported Lineage Scenarios

<table><thead><tr><th width="291">Supported Scenario</th><th>Details</th></tr></thead><tbody><tr><td>DAG → Task → Table Lineage</td><td>Lineage extraction across DAGs, tasks, and database tables</td></tr><tr><td>SQL-based ETL Workflows</td><td>Lineage for SQL-driven ETL processing</td></tr><tr><td>Table-to-Table Lineage</td><td>Source-to-target table relationships</td></tr><tr><td>Column-level SQL Lineage</td><td>Column-level lineage generated from SQL statements</td></tr><tr><td>Join Operations</td><td>Lineage across SQL joins</td></tr><tr><td>Column Aliases</td><td>Resolution of aliased columns</td></tr><tr><td>Aggregations</td><td>Lineage for aggregation functions</td></tr><tr><td>CASE Statements</td><td>Lineage across conditional SQL expressions</td></tr><tr><td>Common Table Expressions (CTEs)</td><td>Lineage through CTE-based queries</td></tr><tr><td>Subqueries</td><td>Lineage for nested SQL queries</td></tr><tr><td>UNION / INTERSECT Operations</td><td>Lineage across SQL set operations</td></tr><tr><td>Multi-schema Lineage</td><td>Lineage across multiple schemas</td></tr><tr><td>Cross-database Lineage</td><td>Lineage across supported databases</td></tr><tr><td>Hive Workflows</td><td>SQL lineage for Hive tasks</td></tr><tr><td>Snowflake Workflows</td><td>SQL lineage for Snowflake tasks</td></tr><tr><td>SQL Server Workflows</td><td>SQL lineage for SQL Server tasks</td></tr><tr><td>MySQL Workflows</td><td>SQL lineage for MySQL tasks</td></tr><tr><td>View-based Lineage</td><td>Lineage across SQL views</td></tr></tbody></table>

{% hint style="info" %}
Lineage extraction is based on SQL statements available within Airflow task definitions and the metadata available through configured database connections.
{% endhint %}

## Partial or Limited Coverage

Certain scenarios provide partial lineage coverage due to SQL complexity, runtime behavior, or metadata limitations.

### Scenarios

<table><thead><tr><th width="237">Scenario</th><th width="247">Limitation Description</th><th width="308">Recommendation</th></tr></thead><tbody><tr><td>Dynamic SQL</td><td>Runtime variables and parameterized SQL are not resolved.</td><td>Use static SQL where possible.</td></tr><tr><td>PythonOperator</td><td>Supported only when SQL is embedded within the task JSON.</td><td>Embed SQL directly in the task definition.</td></tr><tr><td>BashOperator</td><td>Shell scripts are not parsed for lineage extraction.</td><td>Execute SQL through supported SQL-based tasks.</td></tr><tr><td>BigQuery, Oracle, and Amazon Redshift Operators</td><td>Require explicit SQL dialect configuration.</td><td>Configure the appropriate SQL dialect in CrawlerOption.supportingDialects.</td></tr><tr><td>Stored Procedures</td><td>Procedure calls are detected, but internal procedure logic is not parsed.</td><td>Validate procedure-level lineage separately where required.</td></tr><tr><td>TaskGroups / SubDAGs</td><td>Hierarchical workflow handling is limited.</td><td>Validate lineage for complex workflow hierarchies.</td></tr><tr><td>Jinja Templates</td><td>Template rendering is not supported during lineage extraction.</td><td>Use resolved SQL definitions where available.</td></tr><tr><td>Dataset Lineage Status</td><td>Lineage may remain incomplete when SQL extraction fails.</td><td>Verify SQL availability and metadata accessibility.</td></tr></tbody></table>

{% hint style="warning" %}
Runtime-generated SQL, template rendering, and unavailable SQL metadata may reduce lineage completeness and column-level accuracy.
{% endhint %}

## Unsupported Scenarios

The connector does not support lineage extraction for non-SQL processing or runtime-generated execution patterns.

### Unsupported Lineage

| Not Supported                               | Description                                           | Recommendation                                                           |
| ------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------ |
| Python, Pandas, and PySpark Transformations | Non-SQL transformation logic is not parsed.           | Document lineage manually or use supported SQL workflows.                |
| SparkSubmitOperator                         | Spark job lineage is not extracted.                   | Capture lineage using supported Spark integrations where applicable.     |
| File-based ETL                              | CSV, Parquet, and JSON processing is not analyzed.    | Use supported database-based ingestion workflows.                        |
| API-based Processing                        | API execution lineage is not supported.               | Document API lineage separately.                                         |
| Kafka and Streaming Workflows               | Streaming lineage is not available.                   | Capture lineage using supported streaming integrations where applicable. |
| Airflow Variables                           | Variable substitution is not resolved during parsing. | Use static SQL definitions where possible.                               |
| XCom Data Movement                          | Data exchanged through XCom is not tracked.           | Document dependencies manually where required.                           |
| External Shell Scripts                      | External script execution is not parsed.              | Execute SQL through supported Airflow SQL tasks.                         |
| Custom Operators Without SQL Exposure       | Operators without accessible SQL cannot be analyzed.  | Use SQL-based operators or document lineage manually.                    |

{% hint style="info" %}
Unsupported scenarios do not generate lineage and may appear as disconnected nodes within lineage visualization.
{% endhint %}

## Current Functional Status

This section summarizes the current lineage capabilities available for the Apache Airflow connector.

<table><thead><tr><th width="216">Status Area</th><th>Details</th></tr></thead><tbody><tr><td>Overall Coverage</td><td>Strong coverage for SQL-based Airflow workflows</td></tr><tr><td>Lineage Depth</td><td>Table-level and column-level SQL lineage</td></tr><tr><td>Supported Inputs</td><td>DAGs, SQL-based tasks, static SQL queries, views, and supported SQL dialects</td></tr><tr><td>Functional Scope</td><td>SQL lineage extraction across ETL orchestration workflows</td></tr><tr><td>Limitation Areas</td><td>Dynamic SQL, runtime variables, templates, Python transformations, file-based processing, and non-SQL operators</td></tr><tr><td>Resulting Output</td><td>Reliable lineage for standard SQL-based Airflow workflows with partial coverage for advanced orchestration scenarios</td></tr></tbody></table>

{% hint style="warning" %}
The Apache Airflow connector provides strong lineage coverage for SQL-based DAGs and ETL workflows, including table-level and column-level lineage across supported SQL dialects. Advanced scenarios involving dynamic SQL, runtime variables, Python transformations, templates, and non-SQL processing have partial or unsupported coverage and may require manual validation.
{% endhint %}

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ovaledge.com/release8.2/connectors/connector-repositories/etl-tool/apache-airflow/apache-airflow-lineage.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.