> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/matillion/matillion-lineage.md).

# Matillion - Lineage

This article outlines the lineage coverage, configuration settings, metadata handling, supported scenarios, component behaviors, process flow, and known gaps for lineage extraction in the Matillion. The lineage process captures how data moves from source tables and files, including Amazon S3, through the Matillion ETL jobs and transformations to target tables. It supports both table-level and column-level lineage, providing end-to-end visibility into data movement and transformation logic across the Matillion workflows.

### Lineage Configuration Requirements

Accurate lineage extraction in Matillion depends on proper configuration and the availability of metadata. The required settings and access must be configured correctly to ensure that Matillion jobs, SQL queries, source systems, and target objects are parsed and interpreted accurately during lineage generation.

#### Configuration Requirements Table

<table><thead><tr><th width="239.79998779296875">Configuration</th><th>Required Detail</th></tr></thead><tbody><tr><td>Matillion Job Metadata</td><td>Matillion job metadata (JSON) must be available in the source code or exported metadata</td></tr><tr><td>Database Access</td><td>Database access to Snowflake / Redshift is required for table and column resolution</td></tr><tr><td>S3 Access</td><td>S3 access is required for file-to-table lineage</td></tr><tr><td>SQL Queries</td><td>SQL queries must be available in the Matillion component parameters</td></tr><tr><td>SQL Dialect Definition</td><td>Supporting SQL dialects must be configured for SQL query parsing</td></tr><tr><td>Metadata Availability</td><td>Source and target database metadata must be pre-crawled in OvalEdge</td></tr></tbody></table>

{% hint style="warning" %}
Incorrect or missing configuration may prevent the connector from identifying the correct lineage path.
{% endhint %}

### Lineage Components

| Component                 | Availability |
| ------------------------- | :----------: |
| SQL Query Components      |       ✅      |
| SQL Script Components     |       ✅      |
| Transformation Components |       ✅      |
| S3 Load Components        |       ✅      |
| Views / Tables            |       ✅      |
| JOIN / CTE / Subqueries   |       ✅      |
| UNION operations          |       ✅      |
| Stored Procedures         |      ⚠️      |
| Matillion Variables       |      ⚠️      |
| Dynamic SQL               |       ❌      |
| Truncate Operations       |       ❌      |
| Non-SQL Components        |       ❌      |
| SQL Query Components      |       ✅      |
| S3 file to table lineage  |       ✅      |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

## Supported Use Cases

The connector supports several datasets and visualization lineage scenarios across different components of Matillion. These use cases describe areas where lineage extraction functions as expected.

### Supported Lineage Scenarios

| Supported                           | Details                                                                            |
| ----------------------------------- | ---------------------------------------------------------------------------------- |
| SQL-Based ETL Transformations       | Lineage extraction for SQL-based transformations within Matillion jobs             |
| Multi-Component Job Lineage         | Parent-child lineage tracking across multiple connected components                 |
| Table-to-Table Lineage              | Lineage mapping between source and target tables across supported databases        |
| Column-Level Lineage                | Column-level lineage generated through SQL parsing and transformation analysis     |
| File-to-Table Lineage               | Lineage extraction from S3 files to target database tables                         |
| Complex SQL Transformations         | Support for JOINs, CTEs, nested queries, and multi-step SQL transformations        |
| Multi-Step Transformation Pipelines | Lineage tracking across sequential transformation stages within a job              |
| Stored Procedures                   | Partial support for lineage extraction from stored procedure-based transformations |
| Variable-Based Transformations      | Partial support for variable-driven transformation logic                           |
| Dynamic SQL                         | Dynamic SQL-based lineage extraction is not supported                              |
| Non-SQL Components                  | Lineage extraction is not supported for non-SQL-based components                   |

{% hint style="info" %}
Column transformation lineage is supported only for rename and add-column actions.
{% endhint %}

## Partial or Limited Coverage

Certain scenarios require partial coverage due to limitations in metadata, query formats, or processing complexity. These areas may produce incomplete lineage results.

### Scenarios

| Scenario                       | Limitation Description                                                       |
| ------------------------------ | ---------------------------------------------------------------------------- |
| Stored Procedures              | Lineage extraction for stored procedures has limited support                 |
| Matillion Variables            | Only basic variable cleanup and resolution are supported                     |
| Complex Expressions            | Complex expressions and nested logic may require manual validation           |
| Window Functions and UDFs      | Window functions and user-defined functions (UDFs) may not be fully resolved |
| Non-SQL Component Dependencies | Component dependencies without SQL are not fully captured                    |
| Component Parameter Coverage   | Only specific component parameters are processed for lineage extraction      |
| S3 File Extraction             | S3 file lineage extraction relies on regex-based parsing                     |

{% hint style="warning" %}
Incomplete metadata or encrypted data sources can prevent lineage creation entirely.
{% endhint %}

## Unsupported Scenarios

The connector does not support lineage extraction for certain components and modeling features due to either a lack of accessible metadata or non-SQL-based logic.

### Unsupported Lineage

| Not Supported                     | Description                                                                |
| --------------------------------- | -------------------------------------------------------------------------- |
| Dynamic SQL                       | Runtime-generated or dynamically constructed SQL queries are not supported |
| Non-SQL Transformation Components | Lineage is not generated for non-SQL-based transformation components       |
| Real-Time / Streaming Pipelines   | Real-time and streaming pipeline lineage is not supported                  |
| Non-S3 File Sources               | File-to-table lineage is supported only for S3-based sources               |
| Truncate Operations               | TRUNCATE-based operations are not captured in lineage                      |
| Complex Variable Substitution     | Advanced variable substitution patterns are not fully supported            |

{% hint style="info" %}
Unsupported scenarios will not produce lineage and may appear disconnected in lineage visualization.
{% endhint %}

## Current Functional Status

This section outlines the current state of lineage coverage supported by the Matillion connector, including its available capabilities and limitations.

| Status Area      | Details                                                                                                                                        |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Overall Coverage | Partial coverage across Matillion ETL jobs, SQL transformations, and S3 ingestion                                                              |
| Lineage Depth    | Table-level and column-level lineage are supported through SQL parsing                                                                         |
| Supported Inputs | Matillion job metadata, SQL queries, Snowflake, Redshift, and S3 sources                                                                       |
| Functional Scope | Lineage extraction supports standard SQL-based ETL workflows and multi-component jobs                                                          |
| Limitation Areas | Dynamic SQL, non-SQL transformations, complex variable substitution, and incomplete component dependency tracking                              |
| Resulting Output | Lineage is generated for most supported ETL scenarios, with partial or limited mapping for unsupported components and dynamic processing logic |

{% hint style="warning" %}
Coverage remains partial because certain expressions, relationships, and metadata fields are not processed fully.
{% endhint %}

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/matillion/matillion-lineage.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.