Data Quality Functions
The Data Quality Functions module manages system and custom functions designed for various data objects, including Tables, Table Columns, Files, and File Columns. Users can navigate and manage these functions using a Tree View structure.
Selecting a Data Object Type
From the left menu, click on a specific Data Object Type.
Two options will appear:
System
Custom
Viewing System Functions
Selecting the System option displays all predefined system functions associated with the chosen data object type.
These functions are read-only and cannot be edited.
Viewing Custom Functions
Selecting the Custom option displays all user-defined custom functions linked to the selected data object type.
Custom functions are editable, allowing users to modify existing functions or create new ones under the selected data object type.
Adding Custom Functions
Users can add custom functions for the required object type along with the default dimension.
Object Type: Displays the default object type. It can be one of the following: Tables, Table Columns, Files, or File Columns.
Function Name: Enter the name of the function.
Function Description: Describe the function’s purpose, how it operates, and the type of output it generates.
Dimension: Select a dimension from the dropdown, such as Uniqueness, Completeness, or others.
Function Details
When a Data Object Type is selected from the left menu, the corresponding system and custom functions will appear on the right side of the page. The displayed information includes Function Name, Description, Type, Status, etc.
Detailed Function Information
Selecting a specific function name will provide detailed information, including Input and Success Criteria, Queries, Function Help, and History Details.
Function Details
The Function Details section presents information about each data quality function, enabling users to evaluate its capabilities and relevance.
Function Details Includes:
Object Type: Displays whether it relates to Tables, Table Columns, Files, or File Columns.
Function Name: Displays the name of the function.
Function Description: It explains what the function does, how it operates, and the type of output it generates, helping users understand its purpose.
Dimension: Indicates aspects of the function, such as Uniqueness, Completeness, etc.
Execution Metrics: Indicates how the result values are presented, such as Count or Percentage
Support Remediation Center: Specifies whether the function supports a remediation center.
Support Filter: Indicates if the function supports data filtering.
Support Incremental Data Filtering: Indicates whether the function supports incremental data filtering, where rules evaluate only newly added or updated records by specifying an incremental tracking column such as last_updated or created_at.
Rule Query Return Type: Specifies the expected output type of the function’s rule query.
Object Type: Displays whether it relates to Tables, Table Columns, Files, or File Columns.
Function Name: Displays the name of the function.
Function Description: It explains what the function does, how it operates, and the type of output it generates, helping users understand its purpose.
Dimension: Indicates aspects of the function, such as Uniqueness, Completeness, etc.
Execution Metrics: Indicates how the result values are presented, such as Count or Percentage
Remediation Center Support: Specifies whether the function supports a remediation center.
Data Filter Support: Indicates if the function supports data filtering.
Input and Success Criteria Details
The Input and Success Criteria Details section outlines the functionality and options available for data quality functions:
Input Details
Support Input: Indicates whether the function allows users to provide input values.
Yes – Indicates that the function allows user-defined input values.
No – Indicates that the function does not accept user input values.
Input Criteria Label: Specifies the label or name that is displayed to users for entering input criteria. This helps identify the purpose of the input field in the function configuration.
Input Operators: Defines the comparison or logical operators available for the function (e.g., =, !=, >, <). These operators determine how the input values are evaluated against the data.
Input Value Data Type: Indicates the type of data that can be provided as input (e.g., Number, Date, Text). This ensures that only valid and compatible input values are accepted by the function.
Success Criteria Details
Support Success Criteria: Indicates whether the function allows setting success criteria to validate the outcome of the rule.
Yes – The function allows users to define success criteria to evaluate results after execution.
No – The success criteria do not apply to this function.
Example: A data quality rule that checks for null values and validates if less than 5% are null would have Support Success Criteria: Yes.
Success Criteria Validations: Defines the validation method used to assess rule success. Example:
Percent – Validation is based on percentage (e.g., less than 5% null values).
Count – Validation is based on the number of records (e.g., fewer than 100 null values). Example Value: Percent
Success Criteria Label Before: Defines the text that appears before the value in the success message. Example: "The data density" – Used in the sentence: "The data density is greater than 95%".
Success Criteria Label After: Defines the text that appears after the value in the success message. Example: "of the table" – Used in the sentence: "The data density is greater than 95% of the table."
Success Criteria Operators: Specifies comparison operators used to evaluate the criteria. Examples:
Equal To – Value must exactly match (e.g., = 100).
Not Equal To – Value must differ (e.g., ≠ 0).
Success Value Data Type: Defines the data type for success criteria values. Examples:
Number – Used for percentages or counts (e.g., 95%, 100 records).
Date – Used when the criteria involve date comparisons.
Text – Used when matching specific strings.
Example Value: Number
Note: These settings are editable only for custom-defined functions. For system-defined functions, the input and success criteria fields are displayed in read-only mode.
Queries
The Query Tab displays queries associated with a specific connector, including the supported data types for the function. These queries are used to execute data quality rules.
Query Types
Rule Query: Fetches results after executing the data quality rule.
Statistics Query: Retrieves the passed row count, failed row count, and total row count for the table or file where the data quality rule is applied.
Failed Values Query: Fetches the failed values after executing the data quality rule.
Custom Functions: For custom functions, users can add or edit queries for the required connector along with the supported data types.
To add a query for a custom function:
Click the + Icon. This opens the Add Connector Queries pop-up window.
Select the Connector
Use the dropdown to select the connector. The dropdown lists all connectors configured in OvalEdge.
Additionally, the user can enter the Connector Name in the Select the Connector field.
Enter Queries
Rule Query: Define the query to fetch results after executing the data quality rule.
Example: SELECT SUM(CASE WHEN ($INPUTQUERY) THEN 1 ELSE 0 END) AS result
FROM $TABLE WHERE 1=1 $FILTER $ORDERBY
Rule Query Return Type: Specify the return type for the Rule Query.
Statistics Query: Enter the query to retrieve counts for passed rows, failed rows, and total rows.
Example: SELECT COUNT(*) AS Totalcount,
SUM(CASE WHEN ($INPUTQUERY) THEN 1 ELSE 0 END) AS Passedcount,
SUM(CASE WHEN ($INPUTQUERY) THEN 0 ELSE 1 END) AS Failedcount
FROM $TABLE
WHERE 1=1 $FILTER $ORDERBY
Failed Values Query: Define the query to fetch failed values after rule execution.
SELECT $SELECTCOLUMNS
FROM $TABLE
WHERE (CASE WHEN ($INPUTQUERY) THEN 0 ELSE 1 END) = 1 $FILTER $ORDERBY
Specify Supported Data Types
Enter the data types supported by the function.
Example: INT, BIGINT, SMALLINT, DECIMAL, FLOAT, DOUBLE, TINYINT, MEDIUMINT, YEAR, BIT, BINARY
Once all fields are completed, save the configuration to associate the queries with the selected connector.
Edit Query
To edit the connector queries
Click on the Connector Name column
Click the Edit icon. This will open the Edit Connector Queries pop-up window
In Edit Connectors Queries users can modify the Rule Query, Statistics Query, Failed Values Query, and specify the Supported Data Types.
Note: The Connector Name is not editable and only the functions which are in Draft status can be edited. Once the function is published, users can no longer add or edit the queries.
After making the necessary changes, save them to apply the updates.
Functional Help
The Function Help section provides a detailed explanation of the function that helps users understand the function's purpose and how to apply it effectively.
It includes:
The formula used for calculations.
A step-by-step example with sample data to demonstrate the function's usage and application.
Examples of Success Criteria and Failed Criteria to illustrate how the function meets or does not meet the required conditions.
Users can define Function Help for custom functions to explain their usage, functionality, and behavior. This includes:
How to use the function.
How the function works, with examples and formulas used for calculations.
The content provided in the Function Help tab is displayed when users add a rule and select the function. Clicking on the Function Help shows these details, enabling users to understand and apply the function effectively to data quality rules.
History
All changes made to Function Details, Input & Success Criteria Details, Queries, and Function Help are tracked and displayed in the History tab. The tab logs the username, date, and timestamp of each activity, such as status changes or updates.
System Functions:
The History tab tracks function status changes and query updates.
Each entry includes the username and the date and time of the change.
Custom Functions:
The History tab displays all updates related to Function Name, Description, Example, Input Conditions, Success Criteria Conditions, Queries, Function Help etc.
Each entry includes the username and the date and time of the change.
Last updated
Was this helpful?