> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/databricks/databricks-lineage.md).

# Databricks – Lineage

This article outlines the lineage coverage, configuration requirements, supported components, transformation types, use cases, and known limitations of the Databricks connector. The Databricks connector provides automated lineage extraction from notebooks by parsing SQL queries embedded in notebook commands. It supports lineage extraction across SQL-based data transformations, Delta Lake operations, and multi-command notebook pipelines. The connector provides table-level and column-level lineage for supported SQL transformations, enabling traceability of data flows across Databricks environments.

## Lineage Configuration Requirements

Accurate lineage extraction depends on the availability of notebook source code, Databricks workspace connectivity, SQL queries, and source and target table metadata. The required SQL dialect configurations and dataset dependencies must be available before lineage generation.

### Configuration Requirements Table

<table data-search="false"><thead><tr><th width="210.48828125">Configuration</th><th>Required Detail</th></tr></thead><tbody><tr><td>Notebook Source Code</td><td>Notebook source code must be available in JSON format for metadata extraction and lineage parsing.</td></tr><tr><td>Databricks Workspace Connection</td><td>The Databricks workspace connection must be configured and accessible.</td></tr><tr><td>SQL Queries</td><td>SQL queries must be present in notebook commands for lineage extraction.</td></tr><tr><td>SQL Dialect</td><td>Supporting SQL dialects must be configured to enable SQL query parsing.</td></tr><tr><td>Parent Dataset</td><td>The parent dataset must exist to provide the required lineage context.</td></tr><tr><td>Source and Target Tables</td><td>Source and target tables referenced in notebook SQL queries must be available for lineage resolution.</td></tr></tbody></table>

{% hint style="warning" %}
Missing notebook source code, inaccessible workspace connections, unavailable SQL dialect configurations, or missing source and target tables may result in incomplete lineage extraction or unresolved source and target mappings.
{% endhint %}

## Lineage Components

The Databricks connector supports lineage extraction across notebooks, SQL commands, Delta tables, and supported SQL-based transformations. Python notebooks provide partial support when SQL queries are executed through `spark.sql()`. The level of lineage coverage depends on the notebook language, query structure, and transformation type.

### Component Support Matrix

<table data-search="false"><thead><tr><th>Component / Feature</th><th align="center">Availability</th></tr></thead><tbody><tr><td>Notebooks (JSON)</td><td align="center">✅</td></tr><tr><td>SQL Commands</td><td align="center">✅</td></tr><tr><td>Python (<code>spark.sql</code>)</td><td align="center">⚠️</td></tr><tr><td>Scala / R Notebooks</td><td align="center">❌</td></tr><tr><td>Delta Tables</td><td align="center">✅</td></tr><tr><td>Temporary Views</td><td align="center">⚠️</td></tr><tr><td>Dynamic SQL</td><td align="center">❌</td></tr><tr><td>PySpark DataFrame API</td><td align="center">❌</td></tr></tbody></table>

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

## Column Creation Support

The Databricks connector supports column-level lineage for direct column mappings, aliases, expressions, calculations, and aggregations within supported SQL transformations. Window functions receive partial support and may not provide complete column-level lineage.

### Column Creation from Lineage

| Transformation Type          | Supported |
| ---------------------------- | :-------: |
| Direct Column Mapping        |     ✅     |
| Column Aliasing              |     ✅     |
| Expressions and Calculations |     ✅     |
| Aggregations                 |     ✅     |
| Window Functions             |     ⚠️    |

{% hint style="warning" %}
Column-level lineage coverage depends on the SQL transformation logic. Window functions and complex SQL constructs may result in incomplete column-level mappings.
{% endhint %}

## Supported Use Cases

The Databricks connector supports lineage extraction across SQL-based notebooks, table transformations, Delta Lake operations, and multi-step notebook pipelines. It also supports Python notebooks containing SQL queries executed through `spark.sql()`.

### Supported Lineage Scenarios

<table data-search="false"><thead><tr><th width="230.4609375">Supported Scenario</th><th>Details</th></tr></thead><tbody><tr><td>Databricks Notebook → Table Lineage</td><td>Lineage extraction from Databricks notebooks to the tables referenced in supported SQL commands.</td></tr><tr><td>Table → Table Lineage</td><td>Lineage extraction across source and target tables referenced in SQL queries.</td></tr><tr><td>Column → Column Lineage</td><td>Column-level lineage across supported SQL transformations.</td></tr><tr><td>Delta Lake Operations</td><td>Lineage extraction for supported Delta Lake operations.</td></tr><tr><td>Multi-step Notebook Pipelines</td><td>Lineage extraction across multiple SQL commands within notebook pipelines.</td></tr><tr><td>Python Notebooks with Embedded SQL</td><td>Lineage extraction from SQL queries executed through <code>spark.sql()</code> in Python notebooks.</td></tr><tr><td>CREATE TABLE / CREATE VIEW Transformations</td><td>Lineage extraction for supported SQL statements that create tables and views.</td></tr></tbody></table>

{% hint style="info" %}
Lineage extraction depends on the availability of notebook source code in JSON format, supported SQL commands, and the required source and target table metadata.
{% endhint %}

## Partial or Limited Coverage

Certain notebook operations and SQL transformation patterns provide partial lineage coverage. Limitations primarily apply to Python notebooks, window functions, data modification operations, temporary views, external tables, and complex SQL constructs.

### Scenarios

<table data-search="false"><thead><tr><th width="230.43359375">Scenario</th><th>Limitation Description</th></tr></thead><tbody><tr><td>Python Notebooks</td><td>Only SQL queries executed through <code>spark.sql()</code> are parsed. Other Python-based processing logic is not covered.</td></tr><tr><td>Window Functions</td><td>Window functions are parsed with partial handling and may not provide complete lineage.</td></tr><tr><td>UPDATE / MERGE Operations</td><td>Column-level lineage may be incomplete for UPDATE and MERGE operations.</td></tr><tr><td>Temporary Views</td><td>Temporary views may not persist fully in lineage.</td></tr><tr><td>External Tables</td><td>Lineage resolution depends on the availability of the required connections and SQL dialect configurations.</td></tr><tr><td>Complex SQL Constructs</td><td>Complex SQL queries may reduce column-level lineage accuracy.</td></tr></tbody></table>

{% hint style="warning" %}
Partial coverage may result in incomplete lineage paths or reduced column-level lineage accuracy, particularly for Python-based processing, temporary views, and complex SQL transformations.
{% endhint %}

## Unsupported Scenarios

The Databricks connector does not provide lineage coverage for certain notebook languages, non-SQL transformation methods, and runtime-dependent query patterns. These limitations primarily apply to PySpark DataFrame operations, dynamic SQL, Scala and R notebooks, streaming operations, and external file processing without SQL representation.

### Unsupported Lineage

<table data-search="false"><thead><tr><th width="230.54296875">Not Supported</th><th width="304.3828125">Impact</th><th width="252.67578125">Workaround</th></tr></thead><tbody><tr><td>Pure PySpark DataFrame Transformations</td><td>Data transformations performed through the PySpark DataFrame API are not parsed for lineage extraction.</td><td>Use supported SQL-based transformations where applicable.</td></tr><tr><td>Dynamic SQL</td><td>Dynamically generated SQL queries are not resolved during lineage extraction.</td><td>Use statically defined SQL queries where applicable.</td></tr><tr><td>Scala / R Notebook Parsing</td><td>Notebook logic written in Scala and R is not parsed for lineage extraction.</td><td>Use supported SQL commands within notebooks where applicable.</td></tr><tr><td>Streaming / Real-time Lineage</td><td>Lineage extraction for streaming and real-time processing is not supported.</td><td>Document streaming dependencies separately.</td></tr><tr><td>External File Reads Without SQL Representation</td><td>External file processing that does not use SQL queries is not captured in lineage.</td><td>Use supported SQL-based processing where applicable.</td></tr><tr><td>Widget / Parameter-driven Queries</td><td>Queries generated through notebook widgets or parameters are not resolved for lineage extraction.</td><td>Use statically defined SQL queries where applicable.</td></tr></tbody></table>

{% hint style="info" %}
Unsupported notebook operations and processing patterns do not produce complete lineage and may result in disconnected or incomplete lineage paths in lineage visualization.
{% endhint %}

## Current Functional Status

The Databricks connector provides strong lineage coverage across SQL-based notebooks, Delta Lake operations, and embedded SQL transformations. It supports table-level lineage and selected column-level lineage across supported SQL commands and multi-step notebook pipelines.

However, lineage coverage remains partial for Python notebooks containing non-SQL logic, window functions, UPDATE and MERGE operations, temporary views, and complex SQL constructs. Pure PySpark DataFrame transformations, dynamic SQL, Scala and R notebook parsing, streaming and real-time lineage, external file reads without SQL representation, and widget or parameter-driven queries are not supported.

### Functional Coverage Summary

<table data-search="false"><thead><tr><th width="180.01953125">Status Area</th><th width="594.953125">Details</th></tr></thead><tbody><tr><td>Overall Coverage</td><td>Partial coverage across SQL-based Databricks notebooks, Delta Lake operations, and supported SQL transformations.</td></tr><tr><td>Lineage Depth</td><td>Table-level lineage and column-level lineage for supported SQL queries and transformations.</td></tr><tr><td>Supported Inputs</td><td>JSON-based notebooks, SQL commands, Delta tables, and Python notebooks containing SQL queries executed through <code>spark.sql()</code>.</td></tr><tr><td>Functional Scope</td><td>Lineage extraction across supported notebook commands, SQL transformations, Delta Lake operations, and multi-step notebook pipelines.</td></tr><tr><td>Column-level Coverage</td><td>Direct column mappings, aliases, expressions, calculations, aggregations, and partially supported window functions.</td></tr><tr><td>Limitation Areas</td><td>Python notebooks with non-SQL logic, window functions, UPDATE and MERGE operations, temporary views, external tables, and complex SQL constructs.</td></tr><tr><td>Unsupported Areas</td><td>Pure PySpark DataFrame transformations, dynamic SQL, Scala and R notebooks, streaming and real-time lineage, external file reads without SQL representation, and widget or parameter-driven queries.</td></tr><tr><td>Resulting Output</td><td>Lineage for SQL-based notebooks, Delta Lake operations, and supported SQL transformations, with partial coverage for non-SQL and advanced notebook processing scenarios.</td></tr></tbody></table>

{% hint style="warning" %}
The Databricks connector provides lineage extraction across SQL-based notebooks, Delta Lake operations, and embedded SQL transformations, with table-level and column-level lineage support. Coverage remains partial for certain Python notebook operations, window functions, UPDATE and MERGE operations, temporary views, and complex SQL constructs. Pure PySpark DataFrame transformations, dynamic SQL, Scala and R notebook parsing, streaming and real-time lineage, and widget or parameter-driven queries are not supported.
{% endhint %}

***

Copyright © 2025, OvalEdge LLC, Peachtree Corners GA USA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/databricks/databricks-lineage.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
