> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/release8.2/connectors/connector-repositories/other/spline/spline-lineage.md).

# Spline - Lineage

This article outlines the lineage coverage, configuration settings, metadata handling, supported scenarios, component behaviors, process flow, and known gaps for lineage extraction in Spline. The lineage process captures Apache Spark execution metadata to build end-to-end lineage across files, databases, cloud storage platforms, and Spark-based data pipelines. It supports detailed column-level lineage and transformation tracking, enabling visibility into joins, aggregations, projections, and expression-based transformations from source to target systems.

The connector is particularly valuable for Spark-based ETL environments, providing lineage across multiple platforms such as S3, Delta Lake, JDBC-connected systems, and other supported storage and processing technologies.

### Lineage Configuration Requirements

Accurate lineage extraction depends on specific configuration settings and metadata availability. These requirements must be properly configured and available to ensure successful lineage generation and source-target resolution.

#### Configuration Requirements Table

<table><thead><tr><th width="244">Configuration</th><th>Required Detail</th></tr></thead><tbody><tr><td>Spline JSON Payloads</td><td>Valid Spline lineage JSON payloads must be available for processing</td></tr><tr><td>File/Data Lake Lineage Configuration</td><td>SPLINE_LINEAGE_CONNECTION must be configured for file and data lake lineage extraction</td></tr><tr><td>Database Connections</td><td>Source and target database connections must be configured for object resolution</td></tr><tr><td>Connection Priority</td><td>Connection priority for lineage must be defined in connectionsPriorityForLineage</td></tr><tr><td>Schema Metadata Access</td><td>Access to schema and catalog metadata is required</td></tr><tr><td>Security Permissions</td><td>Appropriate security context and permissions must be available for lineage processing</td></tr></tbody></table>

{% hint style="warning" %}
Missing lineage payloads, unresolved connections, or insufficient metadata access may prevent lineage extraction or result in incomplete lineage paths.
{% endhint %}

### Lineage Components

| Component                          | Availability |
| ---------------------------------- | :----------: |
| Spark SQL Jobs                     |       ✅      |
| DataFrame / Dataset API            |       ✅      |
| Source Tables (Hive/Delta/Parquet) |       ✅      |
| Target Tables                      |       ✅      |
| Column-Level Lineage               |       ✅      |
| Joins (Multiple Sources)           |       ✅      |
| Expressions / Derived Columns      |       ✅      |
| Renamed Columns (Aliases)          |       ✅      |
| Aggregations (SUM, COUNT, etc.)    |      ⚠️      |
| Temporary Views                    |       ✅      |
| File Sources (S3/ADLS/HDFS)        |       ✅      |
| File Targets                       |       ✅      |
| Streaming Jobs                     |      ⚠️      |
| UDFs (User Defined Functions)      |       ❌      |
| Stored Procedures                  |       ❌      |
| External Non-Spark Queries         |       ❌      |
| Parameters / Runtime Variables     |       ❌      |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

### Column Creation Support

This section outlines lineage support for column creation and transformation scenarios.

#### Column Creation from Lineage

| Transformation Type           | Availability |
| ----------------------------- | :----------: |
| Direct Column Mapping         |       ✅      |
| Expressions (Derived Columns) |       ✅      |
| Joins (Multi-source Mapping)  |       ✅      |
| Renamed Columns (Aliases)     |       ✅      |
| Aggregations                  |      ⚠️      |
| Calculated Fields             |       ✅      |
| Temporary Columns             |       ✅      |
| Semantic Columns              |       ✅      |

{% hint style="info" %}
The ⚠️ icon indicates partially supported functionality with limited lineage coverage in applicable scenarios.
{% endhint %}

## Supported Use Cases

The connector supports lineage extraction across Spark-based data processing workloads and transformation pipelines. These use cases represent scenarios where lineage extraction functions as expected.

### Supported Lineage Scenarios

<table><thead><tr><th width="245">Supported</th><th>Details</th></tr></thead><tbody><tr><td>Spark Batch ETL Jobs</td><td>Lineage extraction from Spark batch processing workloads</td></tr><tr><td>File-to-Table Lineage</td><td>Tracking lineage from files to database tables</td></tr><tr><td>Table-to-Table Lineage</td><td>Lineage between source and target tables</td></tr><tr><td>Multi-Source Joins</td><td>Column-level lineage across joins involving multiple data sources</td></tr><tr><td>Aggregations and Projections</td><td>Lineage for aggregation and projection operations</td></tr><tr><td>Expression-Based Transformations</td><td>Tracking of derived columns and transformation expressions</td></tr><tr><td>Cross-Platform Pipelines</td><td>Lineage across S3, Delta Lake, JDBC, and other supported platforms</td></tr><tr><td>JDBC-Based Lineage</td><td>Source and target resolution through JDBC-connected systems</td></tr><tr><td>Partitioned Data Processing</td><td>Lineage extraction for partitioned datasets</td></tr><tr><td>Column-Level Lineage</td><td>Detailed source-to-target column mapping with transformation tracking</td></tr></tbody></table>

{% hint style="info" %}
Column mappings and transformation lineage are generated from Apache Spark execution metadata and supported transformation information available in the lineage payload.
{% endhint %}

## Partial or Limited Coverage

Certain scenarios provide only partial lineage coverage due to limitations in execution metadata availability, transformation complexity, or Spark processing behavior.

### Scenarios

<table><thead><tr><th width="245">Scenario</th><th>Limitation Description</th></tr></thead><tbody><tr><td>Filter / Where Conditions</td><td>Filter logic may not be fully represented in lineage output</td></tr><tr><td>Window Functions</td><td>Limited visibility into window-based transformations</td></tr><tr><td>Aggregations</td><td>Aggregation lineage may not capture the complete transformation context</td></tr><tr><td>Kafka Sources</td><td>Metadata coverage for Kafka-based sources is limited</td></tr><tr><td>Streaming Pipelines</td><td>Streaming lineage support is limited and may produce incomplete results</td></tr><tr><td>Complex Nested Transformations</td><td>Deeply nested transformation chains may not be fully resolved</td></tr><tr><td>Business Lineage View</td><td>Business-level lineage representation is limited</td></tr><tr><td>Compact Execution View</td><td>Execution details may not be fully represented in compact views</td></tr><tr><td>Union Operations</td><td>Lineage generation may be incomplete for union-based transformations</td></tr></tbody></table>

{% hint style="warning" %}
Advanced Spark processing patterns may result in partial lineage visibility even when lineage extraction succeeds.
{% endhint %}

## Unsupported Scenarios

The connector does not support lineage extraction for certain Spark execution patterns and processing methods due to limitations in available metadata or execution plan visibility.

### Unsupported Lineage

<table><thead><tr><th width="243">Not Supported</th><th>Description</th></tr></thead><tbody><tr><td>Dynamic SQL</td><td>Runtime-generated or dynamically constructed SQL statements</td></tr><tr><td>Spark Streaming Stateful Operations</td><td>Stateful streaming transformations and processing logic</td></tr><tr><td>RDD-Based Transformations</td><td>Lineage from Spark RDD operations</td></tr><tr><td>Real-Time Lineage Processing</td><td>Real-time lineage generation and visualization</td></tr><tr><td>Non-Spline Instrumented Spark Jobs</td><td>Spark jobs executed without Spline instrumentation</td></tr><tr><td>Full UDF Logic Parsing</td><td>Internal logic of User Defined Functions</td></tr><tr><td>External Non-Spark Processing</td><td>Transformations executed outside Spark execution frameworks</td></tr><tr><td>Runtime Variables and Parameters</td><td>Runtime-generated variables used during execution</td></tr></tbody></table>

{% hint style="info" %}
Unsupported scenarios will not generate lineage and may appear disconnected or absent in lineage visualization.
{% endhint %}

## Current Functional Status

This section outlines the present state of lineage coverage supported by the Spline connector based on the available capabilities and limitations.

<table><thead><tr><th width="242">Status Area</th><th>Details</th></tr></thead><tbody><tr><td>Overall Coverage</td><td>Advanced coverage across Spark-based data processing pipelines</td></tr><tr><td>Lineage Depth</td><td>Detailed column-level lineage with transformation tracking</td></tr><tr><td>Supported Inputs</td><td>Spark SQL jobs, DataFrame APIs, files, tables, JDBC sources, and cloud storage platforms</td></tr><tr><td>Functional Scope</td><td>End-to-end lineage extraction from Spark execution metadata</td></tr><tr><td>Limitation Areas</td><td>Streaming workloads, dynamic SQL, UDF internals, union operations, and complex nested transformations</td></tr><tr><td>Resulting Output</td><td>Strong source-to-target lineage with robust column-level traceability and transformation visibility</td></tr></tbody></table>

{% hint style="warning" %}
The connector provides production-ready lineage coverage for most Spark ETL workloads; however, advanced scenarios such as streaming operations, dynamic SQL generation, UDF internals, and certain complex transformations remain partially supported or unsupported.
{% endhint %}

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners, GA, USA.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ovaledge.com/release8.2/connectors/connector-repositories/other/spline/spline-lineage.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.