Matillion - Lineage
This article outlines the lineage coverage, configuration settings, metadata handling, supported scenarios, component behaviors, process flow, and known gaps for lineage extraction in the Matillion. The lineage process captures how data moves from source tables and files, including Amazon S3, through the Matillion ETL jobs and transformations to target tables. It supports both table-level and column-level lineage, providing end-to-end visibility into data movement and transformation logic across the Matillion workflows.
Lineage Configuration Requirements
Accurate lineage extraction in Matillion depends on proper configuration and the availability of metadata. The required settings and access must be configured correctly to ensure that Matillion jobs, SQL queries, source systems, and target objects are parsed and interpreted accurately during lineage generation.
Configuration Requirements Table
Matillion Job Metadata
Matillion job metadata (JSON) must be available in the source code or exported metadata
Database Access
Database access to Snowflake / Redshift is required for table and column resolution
S3 Access
S3 access is required for file-to-table lineage
SQL Queries
SQL queries must be available in the Matillion component parameters
SQL Dialect Definition
Supporting SQL dialects must be configured for SQL query parsing
Metadata Availability
Source and target database metadata must be pre-crawled in OvalEdge
Incorrect or missing configuration may prevent the connector from identifying the correct lineage path.
Lineage Components
SQL Query Components
✅
SQL Script Components
✅
Transformation Components
✅
S3 Load Components
✅
Views / Tables
✅
JOIN / CTE / Subqueries
✅
UNION operations
✅
Stored Procedures
⚠️
Matillion Variables
⚠️
Dynamic SQL
❌
Truncate Operations
❌
Non-SQL Components
❌
SQL Query Components
✅
S3 file to table lineage
✅
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
Supported Use Cases
The connector supports several datasets and visualization lineage scenarios across different components of Matillion. These use cases describe areas where lineage extraction functions as expected.
Supported Lineage Scenarios
SQL-Based ETL Transformations
Lineage extraction for SQL-based transformations within Matillion jobs
Multi-Component Job Lineage
Parent-child lineage tracking across multiple connected components
Table-to-Table Lineage
Lineage mapping between source and target tables across supported databases
Column-Level Lineage
Column-level lineage generated through SQL parsing and transformation analysis
File-to-Table Lineage
Lineage extraction from S3 files to target database tables
Complex SQL Transformations
Support for JOINs, CTEs, nested queries, and multi-step SQL transformations
Multi-Step Transformation Pipelines
Lineage tracking across sequential transformation stages within a job
Stored Procedures
Partial support for lineage extraction from stored procedure-based transformations
Variable-Based Transformations
Partial support for variable-driven transformation logic
Dynamic SQL
Dynamic SQL-based lineage extraction is not supported
Non-SQL Components
Lineage extraction is not supported for non-SQL-based components
Column transformation lineage is supported only for rename and add-column actions.
Partial or Limited Coverage
Certain scenarios require partial coverage due to limitations in metadata, query formats, or processing complexity. These areas may produce incomplete lineage results.
Scenarios
Stored Procedures
Lineage extraction for stored procedures has limited support
Matillion Variables
Only basic variable cleanup and resolution are supported
Complex Expressions
Complex expressions and nested logic may require manual validation
Window Functions and UDFs
Window functions and user-defined functions (UDFs) may not be fully resolved
Non-SQL Component Dependencies
Component dependencies without SQL are not fully captured
Component Parameter Coverage
Only specific component parameters are processed for lineage extraction
S3 File Extraction
S3 file lineage extraction relies on regex-based parsing
Incomplete metadata or encrypted data sources can prevent lineage creation entirely.
Unsupported Scenarios
The connector does not support lineage extraction for certain components and modeling features due to either a lack of accessible metadata or non-SQL-based logic.
Unsupported Lineage
Dynamic SQL
Runtime-generated or dynamically constructed SQL queries are not supported
Non-SQL Transformation Components
Lineage is not generated for non-SQL-based transformation components
Real-Time / Streaming Pipelines
Real-time and streaming pipeline lineage is not supported
Non-S3 File Sources
File-to-table lineage is supported only for S3-based sources
Truncate Operations
TRUNCATE-based operations are not captured in lineage
Complex Variable Substitution
Advanced variable substitution patterns are not fully supported
Unsupported scenarios will not produce lineage and may appear disconnected in lineage visualization.
Current Functional Status
This section outlines the current state of lineage coverage supported by the Matillion connector, including its available capabilities and limitations.
Overall Coverage
Partial coverage across Matillion ETL jobs, SQL transformations, and S3 ingestion
Lineage Depth
Table-level and column-level lineage are supported through SQL parsing
Supported Inputs
Matillion job metadata, SQL queries, Snowflake, Redshift, and S3 sources
Functional Scope
Lineage extraction supports standard SQL-based ETL workflows and multi-component jobs
Limitation Areas
Dynamic SQL, non-SQL transformations, complex variable substitution, and incomplete component dependency tracking
Resulting Output
Lineage is generated for most supported ETL scenarios, with partial or limited mapping for unsupported components and dynamic processing logic
Coverage remains partial because certain expressions, relationships, and metadata fields are not processed fully.
Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA
Last updated
Was this helpful?