Spline - Lineage
This article outlines the lineage coverage, configuration settings, metadata handling, supported scenarios, component behaviors, process flow, and known gaps for lineage extraction in Spline. The lineage process captures Apache Spark execution metadata to build end-to-end lineage across files, databases, cloud storage platforms, and Spark-based data pipelines. It supports detailed column-level lineage and transformation tracking, enabling visibility into joins, aggregations, projections, and expression-based transformations from source to target systems.
The connector is particularly valuable for Spark-based ETL environments, providing lineage across multiple platforms such as S3, Delta Lake, JDBC-connected systems, and other supported storage and processing technologies.
Lineage Configuration Requirements
Accurate lineage extraction depends on specific configuration settings and metadata availability. These requirements must be properly configured and available to ensure successful lineage generation and source-target resolution.
Configuration Requirements Table
Spline JSON Payloads
Valid Spline lineage JSON payloads must be available for processing
File/Data Lake Lineage Configuration
SPLINE_LINEAGE_CONNECTION must be configured for file and data lake lineage extraction
Database Connections
Source and target database connections must be configured for object resolution
Connection Priority
Connection priority for lineage must be defined in connectionsPriorityForLineage
Schema Metadata Access
Access to schema and catalog metadata is required
Security Permissions
Appropriate security context and permissions must be available for lineage processing
Missing lineage payloads, unresolved connections, or insufficient metadata access may prevent lineage extraction or result in incomplete lineage paths.
Lineage Components
Spark SQL Jobs
✅
DataFrame / Dataset API
✅
Source Tables (Hive/Delta/Parquet)
✅
Target Tables
✅
Column-Level Lineage
✅
Joins (Multiple Sources)
✅
Expressions / Derived Columns
✅
Renamed Columns (Aliases)
✅
Aggregations (SUM, COUNT, etc.)
⚠️
Temporary Views
✅
File Sources (S3/ADLS/HDFS)
✅
File Targets
✅
Streaming Jobs
⚠️
UDFs (User Defined Functions)
❌
Stored Procedures
❌
External Non-Spark Queries
❌
Parameters / Runtime Variables
❌
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
Column Creation Support
This section outlines lineage support for column creation and transformation scenarios.
Column Creation from Lineage
Direct Column Mapping
✅
Expressions (Derived Columns)
✅
Joins (Multi-source Mapping)
✅
Renamed Columns (Aliases)
✅
Aggregations
⚠️
Calculated Fields
✅
Temporary Columns
✅
Semantic Columns
✅
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
Supported Use Cases
The connector supports lineage extraction across Spark-based data processing workloads and transformation pipelines. These use cases represent scenarios where lineage extraction functions as expected.
Supported Lineage Scenarios
Spark Batch ETL Jobs
Lineage extraction from Spark batch processing workloads
File-to-Table Lineage
Tracking lineage from files to database tables
Table-to-Table Lineage
Lineage between source and target tables
Multi-Source Joins
Column-level lineage across joins involving multiple data sources
Aggregations and Projections
Lineage for aggregation and projection operations
Expression-Based Transformations
Tracking of derived columns and transformation expressions
Cross-Platform Pipelines
Lineage across S3, Delta Lake, JDBC, and other supported platforms
JDBC-Based Lineage
Source and target resolution through JDBC-connected systems
Partitioned Data Processing
Lineage extraction for partitioned datasets
Column-Level Lineage
Detailed source-to-target column mapping with transformation tracking
Column mappings and transformation lineage are generated from Apache Spark execution metadata and supported transformation information available in the lineage payload.
Partial or Limited Coverage
Certain scenarios provide only partial lineage coverage due to limitations in execution metadata availability, transformation complexity, or Spark processing behavior.
Scenarios
Filter / Where Conditions
Filter logic may not be fully represented in lineage output
Window Functions
Limited visibility into window-based transformations
Aggregations
Aggregation lineage may not capture the complete transformation context
Kafka Sources
Metadata coverage for Kafka-based sources is limited
Streaming Pipelines
Streaming lineage support is limited and may produce incomplete results
Complex Nested Transformations
Deeply nested transformation chains may not be fully resolved
Business Lineage View
Business-level lineage representation is limited
Compact Execution View
Execution details may not be fully represented in compact views
Union Operations
Lineage generation may be incomplete for union-based transformations
Advanced Spark processing patterns may result in partial lineage visibility even when lineage extraction succeeds.
Unsupported Scenarios
The connector does not support lineage extraction for certain Spark execution patterns and processing methods due to limitations in available metadata or execution plan visibility.
Unsupported Lineage
Dynamic SQL
Runtime-generated or dynamically constructed SQL statements
Spark Streaming Stateful Operations
Stateful streaming transformations and processing logic
RDD-Based Transformations
Lineage from Spark RDD operations
Real-Time Lineage Processing
Real-time lineage generation and visualization
Non-Spline Instrumented Spark Jobs
Spark jobs executed without Spline instrumentation
Full UDF Logic Parsing
Internal logic of User Defined Functions
External Non-Spark Processing
Transformations executed outside Spark execution frameworks
Runtime Variables and Parameters
Runtime-generated variables used during execution
Unsupported scenarios will not generate lineage and may appear disconnected or absent in lineage visualization.
Current Functional Status
This section outlines the present state of lineage coverage supported by the Spline connector based on the available capabilities and limitations.
Overall Coverage
Advanced coverage across Spark-based data processing pipelines
Lineage Depth
Detailed column-level lineage with transformation tracking
Supported Inputs
Spark SQL jobs, DataFrame APIs, files, tables, JDBC sources, and cloud storage platforms
Functional Scope
End-to-end lineage extraction from Spark execution metadata
Limitation Areas
Streaming workloads, dynamic SQL, UDF internals, union operations, and complex nested transformations
Resulting Output
Strong source-to-target lineage with robust column-level traceability and transformation visibility
The connector provides production-ready lineage coverage for most Spark ETL workloads; however, advanced scenarios such as streaming operations, dynamic SQL generation, UDF internals, and certain complex transformations remain partially supported or unsupported.
Copyright © 2026, OvalEdge LLC, Peachtree Corners, GA, USA.
Last updated
Was this helpful?