> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/azure-data-factory/azure-data-factory-lineage.md).

# Azure Data Factory - Lineage

This article outlines the lineage coverage, configuration requirements, metadata handling, supported scenarios, transformation behavior, process flow, and known limitations for lineage extraction in Azure Data Factory (ADF). The lineage process captures how data moves across pipelines, datasets, linked services, data flows, activities, triggers, and global parameters by analyzing JSON-based metadata definitions and transformation logic. It supports table-level and column-level lineage for supported activities and transformations, enabling visibility into data movement, dependency analysis, transformation tracking, and operational monitoring across cloud-based integration workflows.

### Lineage Configuration Requirements

Accurate lineage extraction depends on properly configured authentication, metadata access, and pipeline definition visibility. These settings must be configured correctly to ensure the successful parsing of pipelines, activities, datasets, linked services, and transformation mappings.

#### Configuration Requirements Table

<table><thead><tr><th width="218">Configuration</th><th>Required Detail</th></tr></thead><tbody><tr><td>Access Permissions</td><td>Read access to pipelines, data flows, datasets, linked services, triggers, and related Azure resources is required</td></tr><tr><td>Azure API Access</td><td>Service principals or managed identities must be allowed to access Azure Data Factory APIs</td></tr><tr><td>Connection Information</td><td>Connection strings and dataset parameters must be accessible for source and target resolution</td></tr><tr><td>Metadata Extraction</td><td>JSON-based metadata extraction must be enabled for pipelines, data flows, datasets, and linked services</td></tr><tr><td>Dynamic Parameter Handling</td><td>Dataset parameters and dynamic paths must be accessible for lineage resolution</td></tr><tr><td>Cross-Subscription Access</td><td>Bridge or connector configuration is required for remote or cross-subscription environments</td></tr><tr><td>Transformation Parsing</td><td>Data flow transformations and activity references must be accessible for lineage processing</td></tr></tbody></table>

{% hint style="warning" %}
Insufficient permissions, inaccessible datasets, unresolved parameters, restricted linked services, or missing connection information may prevent lineage extraction or result in incomplete lineage mapping.
{% endhint %}

### Lineage Components

The connector supports lineage extraction across pipelines, activities, control flow components, and mapping data flow transformations.

#### Activity and Pipeline Component Support

| Component                     | Availability |
| ----------------------------- | :----------: |
| Copy Activity                 |       ✅      |
| Execute Data Flow             |       ✅      |
| Execute Pipeline              |       ✅      |
| Lookup Activity               |       ✅      |
| Stored Procedure Activity     |       ✅      |
| ForEach Activity              |       ✅      |
| If Condition Activity         |       ✅      |
| Switch Activity               |       ✅      |
| Until Activity                |       ✅      |
| Script Activity               |       ✅      |
| Databricks Notebook Activity  |       ✅      |
| Execute SSIS Package Activity |       ✅      |
| Set Variable Activity         |      ⚠️      |
| Filter Activity               |      ⚠️      |
| Wait Activity                 |      ⚠️      |
| Web Activity                  |      ⚠️      |
| Get Metadata Activity         |      ⚠️      |
| Delete Activity               |      ⚠️      |
| Validation Activity           |      ⚠️      |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

#### Mapping Data Flow Transformation Support

| Transformation    | Availability |
| ----------------- | :----------: |
| Source            |       ✅      |
| Sink              |       ✅      |
| Derived Column    |       ✅      |
| Select            |       ✅      |
| Filter            |       ✅      |
| Join              |       ✅      |
| Union             |       ✅      |
| Lookup            |       ✅      |
| Aggregate         |       ✅      |
| Sort              |       ✅      |
| Pivot             |       ✅      |
| Unpivot           |       ✅      |
| Window            |       ✅      |
| Rank              |       ✅      |
| Split             |       ✅      |
| Alter Row         |       ✅      |
| Distinct          |       ✅      |
| Exists            |       ✅      |
| Flatten           |       ✅      |
| Key Generate      |       ✅      |
| Conditional Split |       ✅      |
| Parse             |       ✅      |
| Assert            |      ⚠️      |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

### Column-Level Lineage Coverage

Column-level lineage extraction is supported for standard transformation scenarios where metadata and transformation logic are accessible during parsing.

#### Column Creation and Transformation Coverage

| Transformation Type             | Supported |
| ------------------------------- | :-------: |
| Select and Rename Columns       |     ✅     |
| Join Columns                    |     ✅     |
| Union Columns                   |     ✅     |
| Pivot and Unpivot Columns       |     ✅     |
| Derived Columns and Expressions |     ⚠️    |
| Aggregate Columns               |     ⚠️    |
| Expression Functions            |     ⚠️    |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

{% hint style="info" %}
Column-level lineage is supported for standard source-to-transformation-to-target mappings across supported Azure Data Factory activities and data flow transformations.
{% endhint %}

### Supported Use Cases

The connector supports lineage extraction across standard Azure Data Factory integration workflows, transformation pipelines, and orchestration scenarios.

#### Supported Lineage Scenarios

<table><thead><tr><th width="221">Supported Scenario</th><th>Details</th></tr></thead><tbody><tr><td>Database-to-Database ETL</td><td>Lineage extraction across relational database integrations such as SQL Server, Azure SQL, Oracle, and Snowflake</td></tr><tr><td>File-to-Database Processing</td><td>Lineage extraction from Azure Blob Storage, ADLS CSV/JSON files, and other file-based sources into database targets</td></tr><tr><td>Database-to-File Processing</td><td>Lineage extraction from database queries into Parquet, CSV, or other file-based outputs</td></tr><tr><td>Complex Data Flow Transformations</td><td>Lineage extraction for joins, aggregates, derived columns, pivot/unpivot, and transformation workflows</td></tr><tr><td>Nested Pipeline Dependencies</td><td>Lineage extraction across parent-child pipeline relationships</td></tr><tr><td>Conditional Workflow Processing</td><td>Lineage extraction across If Condition, Switch, Until, and ForEach activities</td></tr><tr><td>Cross-Cloud Integrations</td><td>Lineage extraction across AWS S3, Salesforce, REST integrations, and Azure-based targets</td></tr><tr><td>Column-Level Transformation Lineage</td><td>Column-level lineage for supported transformations including Select, Join, Aggregate, Derived Column, and Pivot/Unpivot</td></tr></tbody></table>

{% hint style="info" %}
Table-level and column-level lineage are supported for standard Azure Data Factory activities, datasets, and transformation workflows where metadata definitions are accessible.
{% endhint %}

### Partial or Limited Coverage

Certain scenarios provide partial lineage coverage due to runtime-generated logic, dynamic metadata resolution, or limitations in transformation visibility.

#### Partial Coverage Scenarios

<table><thead><tr><th width="221">Scenario</th><th>Limitation Description</th></tr></thead><tbody><tr><td>Dynamic SQL Queries</td><td>Parameterized or runtime-generated SQL queries are not fully parsed during lineage extraction</td></tr><tr><td>Complex Nested Expressions</td><td>Highly nested expressions and multi-step transformation logic may not resolve completely</td></tr><tr><td>Schema Drift in Data Flows</td><td>Dynamic schema changes may result in incomplete column-level lineage</td></tr><tr><td>Encrypted Connection Configurations</td><td>Azure Key Vault or encrypted connection values require accessible decrypted configurations</td></tr><tr><td>Databricks Notebook Logic</td><td>Internal notebook transformation logic is not fully visible for lineage extraction</td></tr><tr><td>Custom Activities</td><td>Internal processing logic within custom activities is partially supported</td></tr><tr><td>Parameterized File Paths</td><td>Wildcard paths and partition-based file structures may not resolve fully without pipeline execution metadata</td></tr><tr><td>Variable-Based Runtime Logic</td><td>Runtime-generated values and dynamically assigned variables may provide partial lineage mapping</td></tr></tbody></table>

{% hint style="warning" %}
Incomplete metadata, unresolved parameters, inaccessible encrypted configurations, schema drift, or unsupported runtime logic may prevent complete lineage creation.
{% endhint %}

### Unsupported Scenarios

The connector does not support lineage extraction for unsupported external processing logic, undocumented APIs, or activities that do not expose sufficient metadata for lineage generation.

#### Unsupported Lineage

<table><thead><tr><th width="253">Not Supported</th><th>Description</th></tr></thead><tbody><tr><td>REST and Web Activity Payload Analysis</td><td>Request and response payloads are not analyzed for automatic lineage extraction</td></tr><tr><td>External APIs Without Metadata Contracts</td><td>APIs without structured metadata definitions are unsupported</td></tr><tr><td>Internal SSIS Package Lineage</td><td>SSIS package internals require separate SSIS lineage integration</td></tr><tr><td>Runtime-Generated External Logic</td><td>Externally generated transformation logic without accessible metadata is unsupported</td></tr><tr><td>Unresolved Dynamic Runtime Metadata</td><td>Runtime-generated schemas and inaccessible execution metadata are unsupported</td></tr></tbody></table>

{% hint style="info" %}
Unsupported scenarios do not generate lineage relationships and may appear disconnected in lineage visualization.
{% endhint %}

### Current Functional Status

This section outlines the present lineage coverage supported by the Azure Data Factory connector based on the currently available capabilities and limitations.

<table><thead><tr><th width="211">Status Area</th><th>Details</th></tr></thead><tbody><tr><td>Overall Coverage</td><td>Production-ready coverage for standard Azure Data Factory pipelines, activities, and data flow transformations</td></tr><tr><td>Lineage Depth</td><td>Table-level and column-level lineage are supported for standard workflows</td></tr><tr><td>Supported Inputs</td><td>Pipelines, datasets, linked services, data flows, triggers, activities, and transformation metadata</td></tr><tr><td>Functional Scope</td><td>Lineage extraction supports standard ETL workflows, orchestration logic, and transformation processing</td></tr><tr><td>Partial Coverage Areas</td><td>Dynamic SQL, schema drift, parameterized paths, encrypted configurations, Databricks notebooks, and complex expressions</td></tr><tr><td>Unsupported Areas</td><td>REST payload analysis, undocumented APIs, unresolved runtime metadata, and SSIS package internals</td></tr><tr><td>Resulting Output</td><td>Lineage is generated for supported Azure Data Factory workflows with partial or unavailable mapping in unsupported scenarios</td></tr></tbody></table>

{% hint style="warning" %}
Coverage remains partial for dynamic runtime logic, schema drift, encrypted configuration handling, Databricks notebook internals, parameterized queries, and unsupported external processing because these scenarios do not expose sufficient metadata for complete lineage extraction.
{% endhint %}

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/azure-data-factory/azure-data-factory-lineage.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.