Amazon Redshift - Lineage
This article outlines the lineage coverage, configuration settings, metadata handling, supported scenarios, component behaviors, process flow, and known limitations for lineage extraction in Amazon Redshift.
The Amazon Redshift connector provides automated lineage extraction from Redshift SQL artifacts, including tables, views, and stored procedures. It enables end-to-end visibility into data flow by parsing SQL queries and tracking upstream and downstream dependencies across data warehouse objects. The connector supports both table-level and column-level lineage, helping users understand data movement, perform impact analysis, troubleshoot dependencies, and support governance initiatives.
Lineage Configuration Requirements
Accurate lineage extraction depends on specific configuration settings, access permissions, and SQL metadata availability. Configure the following settings correctly to ensure successful lineage extraction and relationship mapping.
Configuration Requirements Table
Authentication
Configure valid Amazon Redshift connection credentials with access to the required schemas.
Access Requirements
Ensure access to database schemas, tables, and dataset SourceCode metadata. Valid SQL definitions for tables, views, and stored procedures are required for lineage extraction.
SQL Dialect Processing
Redshift-specific SQL parsing and cleanup logic are used to interpret SQL statements and build lineage relationships.
Processing Control
Configure lineage.codes.process.limit to control the volume of SQL code processed during lineage extraction.
Enhanced Parsing
Configure need.hard.coded.values.in.lineage to improve lineage coverage for hard-coded values and SQL expressions.
Self-Lineage Handling
Configure lineage.selfbuild to control self-lineage filtering and processing behavior.
Using explicit table aliases, qualified column names, and properly formatted SQL statements can improve lineage accuracy and column-level mapping.
Missing SourceCode metadata, inaccessible schemas, or invalid SQL definitions may prevent the connector from building complete lineage relationships.
Lineage Components
Tables
✅
Views
✅
Stored Procedures
⚠️
Functions (within SQL)
⚠️
SQL Queries
✅
Column Lineage
⚠️
Temp Tables/Columns
✅
Reports/Charts
❌
Triggers
⚠️
SELECT (standard queries)
✅
INSERT INTO SELECT
✅
CREATE TABLE AS SELECT (CTAS)
✅
View Definition
✅
MERGE / UPSERT
⚠️
Stored Procedure SQL
⚠️
Expressions
⚠️
Aggregations
⚠️
Dynamic SQL (EXEC)
❌
OPENQUERY
❌
Non-lineage DDL (INDEX, OPTIMIZE, etc.)
❌
Direct Mapping
✅
Renamed Columns
✅
Derived Columns (with alias)
⚠️
Expressions
⚠️
Aggregations
⚠️
Temp Columns
✅
Semantic Columns
⚠️
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
Supported Use Cases
The Amazon Redshift connector supports lineage extraction across SQL-based datasets, views, stored procedures, and transformation workflows. These scenarios represent the areas where lineage extraction functions as expected and provides reliable table-level and column-level lineage relationships.
Supported Lineage Scenarios
Table-to-Table Lineage
Supports lineage extraction for CTAS (CREATE TABLE AS SELECT) and INSERT INTO ... SELECT operations to track data movement between source and target tables.
View Lineage
Extracts lineage from view definitions and maps relationships between views, source tables, and dependent views.
Column-Level Lineage
Supports source-to-target column mapping through SQL parsing, alias resolution, and column dependency analysis.
SQL-Based Transformations
Tracks lineage for SQL transformations that use explicit column references and supported SQL constructs.
Stored Procedure Lineage
Supports lineage extraction from stored procedures that contain static SQL statements.
Function-Level Lineage
Supports lineage extraction for functions used within SQL queries when source and target column relationships can be resolved.
Multi-Step SQL Transformations
Tracks lineage across multiple SQL operations and intermediate datasets to build end-to-end lineage relationships.
Temporary Object Handling
Supports lineage generation when temporary objects are used to resolve missing tables or columns during lineage processing.
Column-level lineage accuracy improves when SQL statements use explicit column references, qualified column names, and clear alias definitions.
Stored procedure lineage is supported only for static SQL. Dynamic SQL and runtime-generated statements may result in incomplete lineage extraction.
Partial or Limited Coverage
Certain lineage scenarios have partial coverage due to SQL complexity, metadata availability, or limitations in SQL parsing. These scenarios may produce incomplete lineage relationships or reduced column-level mapping accuracy.
Scenarios
Stored Procedures
Lineage extraction is limited for complex stored procedures that contain mixed SQL patterns, nested logic, or unsupported constructs.
MERGE / UPSERT Statements
Column-level lineage mapping may be partial depending on statement complexity and transformation logic.
Ambiguous Columns
Unqualified column names may reduce the accuracy of column-level lineage and dependency resolution.
Derived Columns Without Aliases
Columns created without explicit aliases may result in incomplete source-to-target column mappings.
Function-Based Transformations
Lineage coverage depends on the ability to parse and resolve function logic within SQL statements.
Column Coordinate Mapping
Column coordinate visualization in the lineage UI depends on the availability of required configuration and metadata.
Temporary Tables and Columns
When metadata is unavailable, temporary tables and columns may be created with an UNKNOWN type, resulting in partial lineage.
Using qualified column names, explicit column aliases, and well-structured SQL statements can improve lineage accuracy and column-level mapping.
Missing SourceCode metadata, complex stored procedure logic, unsupported SQL constructs, or dynamically generated SQL may prevent the connector from building complete lineage relationships.
Unsupported Scenarios
The Amazon Redshift connector does not support lineage extraction for certain dynamic execution patterns, administrative operations, external query mechanisms, and non-SQL assets due to limitations in SQL parsing and metadata availability.
Unsupported Lineage
Dynamic SQL
Runtime-generated SQL statements, including EXEC-based execution patterns, are not supported for lineage extraction.
OPENQUERY and External Query Execution
Queries executed through external query mechanisms are not processed for lineage generation.
Non-Lineage DDL Operations
Administrative operations such as CREATE INDEX, DROP INDEX, ALTER INDEX, OPTIMIZE, and DESC statements are not included in lineage extraction.
REFRESH MATERIALIZED VIEW
Materialized view refresh operations are not tracked as lineage relationships.
Unsupported SQL Syntax
Queries containing unsupported or unrecognized SQL syntax patterns may be excluded from lineage processing.
ETL Package Formats
External ETL package formats, such as .dtsx files, are outside the scope of the Amazon Redshift connector.
Complex Procedural Logic
Advanced procedural logic that cannot be resolved through SQL parsing is not supported for lineage extraction.
Unsupported scenarios do not generate lineage relationships and may appear disconnected or absent in lineage visualizations.
For lineage involving ETL tools, reporting platforms, or external processing frameworks, use the corresponding connector that supports those assets and metadata sources.
Current Functional Status
This section describes the current lineage coverage and functional behavior supported by the Amazon Redshift connector.
Overall Coverage
Partial coverage across Amazon Redshift tables, views, stored procedures, and SQL-based transformation workflows.
Lineage Depth
Supports reliable table-level lineage and partial column-level lineage through SQL parsing and dependency analysis.
Supported Inputs
Tables, views, stored procedures, SQL datasets, and supported SQL transformation statements.
Functional Scope
Lineage extraction works effectively for standard SQL operations, including CTAS, INSERT INTO ... SELECT, view definitions, and supported SQL transformations.
Limitation Areas
Dynamic SQL, complex stored procedures, ambiguous column references, unsupported SQL syntax patterns, and certain advanced transformation scenarios.
Resulting Output
Produces reliable lineage for most Redshift SQL artifacts and transformation workflows, with partial mapping in complex or unsupported scenarios.
The Amazon Redshift connector provides strong lineage capabilities for tables, views, and SQL-based transformations, including column-level lineage for supported SQL patterns. However, dynamic SQL, complex stored procedures, and ambiguous column mappings may result in incomplete lineage and could require manual validation.
Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA
Last updated
Was this helpful?