Compare Profile Results

The Compare Profile Results feature enables users to compare data profiles from different profiling sessions, providing valuable insights into how data characteristics and distributions have evolved.

Users with the Author role and Metadata Write permissions on a schema can access this feature within the Advanced Tools section of OvalEdge.

Functionality

Compare Profile Results enables users to compare two data tables from separate profiling sessions, highlighting changes in the following areas:

• Column Names: Detect additions, deletions, or renaming of columns.

• Column Properties: Identify changes in column position, name, or data type.

• Column Attributes: Analyze modifications to column characteristics, such as nullability and uniqueness constraints.

• Column Statistics: Compare key statistics—such as null counts, distinct counts, top values, and minimum and maximum values—to observe shifts in data distribution.

The comparison output is displayed in a user-friendly grid format that clearly and concisely highlights changes. The grid includes:

• Change Counts: Displays the number of changes for each statistic (e.g., null count, distinct count).

• Popularity and Importance Scores: Shows changes in these scores, reflecting shifts in data relevance or usage patterns.

Benefits

• Track Data Evolution: Monitor how data changes over time. Detect trends and shifts in characteristics and make informed decisions based on a comprehensive view of data evolution.

• Impact Analysis: Identify potential downstream effects of changes to the data profile. Proactively address issues to ensure data consistency and reliability.

• Data Quality Assurance: Detect discrepancies and anomalies introduced during transformations, helping maintain high data quality standards within OvalEdge.

• Data Integration and Transformation: Pinpoint inconsistencies between datasets during integration, enabling smoother data merging and transformation processes.

• Anomaly Detection: Identify deviations from expected profiles, which may indicate data quality issues, security incidents, or fraudulent activity.

Access Compare Profile Results

To access the Compare Profile Results module:

1. Navigate to Advanced Tools.

2. Select Compare Profile Results.

3. The profile comparison interface appears, allowing users to select and compare profiling sessions.

   1. Source Schema: Select the schema that contains the source table (the reference point for comparison).

   2. Source Table: Choose the specific table within the source schema to analyze.

   3. Source Timestamp: Specify the date associated with the profiling of the source table. This defines the starting point of the comparison.

   4. Target Schema: Select the schema that contains the target table to be compared against the source table.

   5. Target Table: Choose the specific table within the target schema for comparison.

   6. Target Timestamp: Specify the profiling date for the target table. This defines the endpoint of the comparison interval.

4. Once all required details are entered, click "Compare Dataset" to initiate the comparison. The results are displayed in a tabular format for easy review.

Analyze the Compared Results

The comparison results provide insights into various aspects of the two profile dates:

• Source vs. Target: Details from the original profile (Source) are displayed on the left, and the compared profile (Target) is on the right.

• Column-Level Comparison: Individual column names are compared, highlighting any changes in their attributes.

• Profile Date: Shows the dates (previous and latest) chosen for comparison.

• Row Count: Displays the updated row count for each profile.

• Popularity & Importance: Indicates the changes in these scores, potentially reflecting shifts in data relevance or usage patterns.

The comparison table offers detailed information for each column:

• Column Position: Shows the order of a column within its respective table. Differences in the number of columns between source and target are highlighted.

• Column Name: Displays the names of the columns from both tables.

• Column Type: Describes each column's data type (e.g., VARCHAR, INT).

• Statistical Information: Compares key statistical metrics for each column, including:

  • Top Value

  • Null Count

  • Distinct Count

  • Minimum Value

  • Maximum Value