General Advanced Jobs

This release introduces a set of new jobs that enable users to update module- or feature-specific data using advanced algorithms.

Name

Description

EdgiMessagesEligibilityForRecipeAdvanceJob

This advanced job evaluates EDGI messages to determine whether they are eligible for inclusion as recipe steps. It identifies whether each message represents a valid step by analyzing the associated agent type, such as SQL_AGENT, PYTHON_AGENT, or ANALYTIC_AGENT. Based on this evaluation, messages are appropriately marked as steps or non-steps. The job processes messages in configurable batches, allowing efficient handling of large message volumes (e.g., 100 messages per batch).

Attributes:

Attr1 How many messages to process in a batch, e.g., 100

AILineageProcessingAdvanceJob

This advanced job leverages AI-driven parsing to automatically extract and map data lineage across RDBMS, reports, and ETL systems. It intelligently interprets complex transformations, queries, and dependencies to build accurate, end-to-end lineage, enhancing overall lineage completeness.

Transfer Data Flow

This job captures and stores data flow mappings between source and downstream connectors in the backend database. Users can search mappings by entering source and target fields.

Critical Data Element Migration

This advanced job updates Critical Data Element (CDE) values across catalog tables to align with the latest classification standards. It replaces legacy values such as “Critical Data Element” with “Yes” and “Recommended” with “System Recommended.” This ensures consistency in CDE labeling and improves clarity for governance and data classification processes.

DBT to DQ Rule Creation

This advanced job reads dbt test definitions from manifest.json files stored in SourceCode and creates corresponding Data Quality (DQ) Rules in OvalEdge. It automatically maps supported dbt tests, such as unique, not null, accepted values, and relationships, to OvalEdge DQ rules and associates them with the specified source connections. Note: This job creates DQ rules only. Schedule a separate advanced job to process the results of DQ rule execution.

DBT DQ Results Sync

This advanced job synchronizes Data Quality (DQ) execution results from external warehouse tables into OvalEdge. It reads DQ result and failed record tables, maps the results to existing OvalEdge DQ rules, and updates rule execution status and metrics.

Note: Ensure the source result tables are crawled and available in OvalEdge before running this job.

Rebuild Fully Qualified Tag Names

This advanced job rebuilds the full tag path based on the current tag hierarchy and updates how tags appear in filters and tooltips. Master Tags display only the tag name, while child tags display the complete hierarchy path. The job also refreshes Tag Extension data to ensure consistency after hierarchy changes, migrations, or bulk tag updates.

Was this helpful?