> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/release8.2/data-quality/data-quality-rules/creating-data-quality-rule.md). # Creating Data Quality Rule The Author users can create data quality rules by associating multiple data objects. * **Access the Creation Page:** * Navigate to the "Data Quality" module and select the "Data Quality Rule" submodule. * Click the "+" icon to initiate data quality rule creation. * **Select a Function:** * The creation page allows users to choose a data quality function. These functions act as the criteria for evaluating data. OvalEdge offers various built-in functions for common checks like accuracy or completeness. * **Associate Data Objects:** * Once a function is chosen, users can define the rule's scope by associating multiple data objects with it. These objects represent the data users want to assess using the chosen function. Think of them as the specific data points to which the rule will be applied. * By associating multiple objects, users can create a broader rule that evaluates data quality across a group of related items. This approach streamlines data quality checks and ensures consistency within datasets. {% hint style="info" %} The other methods for creating data quality rules are Rule Recommendations, OvalEdge APIs, and Load Metadata from Files. {% endhint %} ## Add Rule The Add Rule enables users to create Data Quality Rules (DQRs) to validate and monitor the quality of data assets across the organization. Users can define validation logic using predefined Data Quality Functions or custom SQL-based conditions, configure rule execution criteria, associate relevant data objects, and set up notifications and remediation actions. When creating a rule, users can: * Select the Object Type to determine the data asset being evaluated. * Choose a Data Quality Function or define custom validation logic. * Configure input parameters, success criteria, and warning criteria. * Associate one or more data objects with the rule. * Define the rule name and purpose. * Assign an appropriate Data Quality Dimension. * Configure execution schedules. * Enable notifications and service request creation for failures. * Assign a Steward responsible for monitoring and managing the rule. After configuration, the rule can be saved and activated to continuously evaluate data quality based on the defined criteria.
### Selecting Object Type When creating a Data Quality Rule, users must first select an Object Type. The Object Type determines the type of data asset against which the rule is evaluated and controls the functions and validation options available during rule configuration. The following Object Types are supported: * **Tables:** Select Tables to create rules that evaluate the quality of an entire table. These rules assess dataset-level characteristics such as row counts, duplicate records, completeness, and other table-level quality metrics. * **Table Columns**: Select Table Columns to create rules that validate data within specific table columns. These rules can be used to identify missing values, invalid formats, duplicate values, data inconsistencies, and other column-level quality issues. * **Files:** Select Files to create rules that assess the quality of an entire file stored in a supported file system or storage location. * **File Columns:** Select File Columns to validate data within specific file columns. These rules help ensure data accuracy, consistency, completeness, and conformity within file-based datasets. * **Codes:** Select Codes when the available Data Quality Functions do not satisfy specific business requirements. This Object Type allows users to define custom SQL queries or validation conditions to implement advanced data quality checks. Code-based rules evaluate the quality of underlying data objects, such as tables and columns, using user-defined logic rather than predefined Data Quality Functions. This approach enables organizations to address complex business scenarios and custom validation requirements that cannot be achieved through out-of-the-box functions. {% hint style="info" %} The available Data Quality Functions, input parameters, and validation options vary based on the selected Object Type. {% endhint %} Selecting the appropriate **Object Type** ensures that the Data Quality Rule is applied to the intended data asset and enables users to perform targeted quality validations based on their business requirements. The selected Object Type also determines the functions and configuration options available for the rule. ### Selecting Function After selecting an Object Type, users must choose a Data Quality Function that defines the validation logic to be applied to the selected data assets.
The list of available functions is dynamically filtered based on the selected Object Type, ensuring that users are presented only with functions applicable to the chosen data asset. Users can browse and select the appropriate function from the available list. Each function includes a brief description to help users understand its purpose and intended use. For additional guidance, users can select See More to access detailed documentation, including function behavior, supported object types, input parameters, success criteria, examples, and use cases. Selecting the appropriate function is essential because it determines: * The type of data quality validation performed. * The input parameters required for execution. * The available success and warning criteria. * The Data Quality Dimension associated with the rule. By choosing the function that best aligns with their business requirements, users can create targeted rules that effectively evaluate and monitor data quality across their data assets. {% hint style="info" %} The functions available for selection vary based on the chosen Object Type and supported data sources. {% endhint %}
### Configuring Function Inputs Each Data Quality Function includes configuration parameters that determine how the rule evaluates the selected data objects. Depending on the function, users can configure Input, Success Criteria, and Warn Criteria. The availability and behavior of these fields vary based on the selected Data Quality Function. #### Input The Input field defines the condition, value, pattern, or threshold that the Data Quality Function uses during rule execution. It is primarily used by functions that evaluate individual values within a column or file. **For example**, in a Regular Expression (Regex) validation rule, the Input field contains the regex pattern against which each value is evaluated. The system then determines how many values satisfy or violate the specified condition. Some functions, such as pattern matching, text validation, date validation, and value comparison checks, require user-defined inputs. However, aggregate functions such as Sum, Average, Minimum, Maximum, or Count evaluate the overall result of a dataset and may not require an input value. Depending on the selected function, users may provide: * Numeric values * Text values * Date values * Comma-separated values * Patterns or expressions {% hint style="info" %} Input requirements vary by function. Refer to the function help documentation for supported inputs and configuration guidelines. {% endhint %} #### Success Criteria Success Criteria defines the condition that determines whether a Data Quality Rule execution is considered successful. Users can specify acceptable thresholds using supported operators and select the appropriate validation mode based on business requirements. **For example:** * A regex validation rule may require at least 95% of values to match the specified pattern. * A sum validation rule may require the calculated sum to be greater than or equal to a specified value. During execution, the system compares the calculated result against the configured Success Criteria. If the result satisfies the defined threshold, the rule execution is marked as Success; otherwise, it is evaluated against the Warn Criteria (if configured) or marked as Failure. #### Validation Modes The following validation modes are supported: * **Count** – Evaluates the actual number of records that satisfy the condition. * **Percent** – Evaluates the percentage of records that satisfy the condition. #### Supported Operators The available operators vary by function and data type and may include: * Equal To * Not Equal To * Greater Than * Greater Than or Equal To * Less Than * Less Than or Equal To * In Between * Not In Between * Matching * Starting With * Ending With * Containing * Is Empty * Is Not Empty * Is Null * Is Not Null #### Warn Criteria Warn Criteria enables users to define an intermediate severity level between Success and Failure for User Defined Functions. This allows organizations to identify potential data quality concerns before they become critical failures. When enabled, the system evaluates rule results using three severity levels: | Condition | Result | | -------------------------------------------- | ------- | | Meets Success Criteria | Success | | Meets Warn Criteria but not Success Criteria | Warn | | Meets neither Success nor Warn Criteria | Failure |
**Warn Criteria helps organizations:** * Detect emerging data quality issues early. * Reduce false-positive failures caused by minor deviations. * Prioritize remediation efforts based on severity. * Improve monitoring and governance of data quality trends. **Configuring Warn Criteria** When Support Warn Criteria is enabled: * The system automatically copies the Success Criteria configuration. * Users can modify the Warn Criteria threshold and operators. * Validation rules ensure that Warn thresholds are less restrictive than Success thresholds. **For example:** | Success Criteria | Warn Criteria | | ---------------- | ---------------- | | Greater Than 95% | Greater Than 90% | | Less Than 5% | Less Than 10% | #### Alert on Warn Users can enable Alert on Warn to receive notifications when a rule result is classified as Warn. **Unlike Failure results:** * Warn results do not generate Service Requests. * Warn-level failed values can be sent to the Data Quality Remediation Center when remediation is enabled. #### Associating Objects The Associated Objects section allows users to select the data assets against which the rule will be executed. Based on the selected Object Type, users can associate: * Tables * Table Columns * Files * File Columns * Code Objects The Add Object dialog enables users to search, select, and associate one or more supported data objects with the rule. {% hint style="info" %} Only objects with data types supported by the selected Data Quality Function can be associated with the rule. Supported data types are listed in the function help documentation. {% endhint %} Associating the correct objects ensures that the rule is executed only against compatible assets and produces accurate validation results. #### Configuring Rule Name Every Data Quality Rule must have a unique name. The Rule Name field allows users to specify a meaningful and descriptive name that clearly identifies the purpose of the rule. A well-defined rule name helps users easily locate, understand, and manage Data Quality Rules across the application. {% hint style="info" %} It is recommended to use naming conventions that reflect the rule objective, target object, or validation criteria to improve discoverability and governance. {% endhint %} #### Configuring Rule Purpose The Purpose field is mandatory and is used to document the business objective, intent, and expected outcome of the Data Quality Rule. It provides context to users about why the rule was created and the data quality issue it is designed to address. Users should enter a clear and concise description that explains: * The business requirement being validated. * The data quality issue or risk being monitored. * The expected outcome of the rule. * Any relevant business context or compliance requirements. Providing a meaningful purpose helps Stewards, Administrators, and other stakeholders understand the rationale behind the rule, supports governance initiatives, and improves the maintainability of Data Quality Rules over time. {% hint style="info" %} Use the Purpose field to describe the business objective of the rule rather than the technical implementation details. This makes the rule easier to understand and manage across teams. {% endhint %} #### Selecting Dimension Data quality dimensions are specific aspects used to evaluate and quantify the quality of data within a dataset. Each Data Quality Function is linked with a default dimension, allowing users to categorize created rules based on that dimension. OvalEdge has eight system-defined dimensions, providing a framework for categorizing data quality rules effectively. * **Accuracy:** This dimension measures the degree of correctness and precision of the data. It focuses on eliminating errors, inconsistencies, or inaccuracies that might be present in the data. * **Completeness:** The completeness dimension evaluates the extent to which all required data is present and available. It ensures that there are no missing values or gaps in the dataset, enabling a comprehensive understanding of the data. * **Consistency:** Consistency examines the uniformity and coherence of data across different sources or time periods. It ensures that data is consistent in terms of format, structure, and meaning, regardless of its origin. * **Timeliness:** Timeliness refers to the relevance of data. It ensures that data is up-to-date and available when needed, enabling informed decision-making and analysis based on the most current information. * **Validity:** The validity dimension assesses the extent to which data conforms to predefined rules or standards. It ensures that data meets the specified criteria and is fit for its intended purpose. * **Uniqueness:** Uniqueness focuses on the absence of duplicate or redundant data records. It ensures that each data record is unique and that there are no unnecessary repetitions, which can affect data analysis and processing. * **Integrity:** The integrity dimension represents the overall accuracy, consistency, and reliability of data throughout its lifecycle. It encompasses multiple aspects of data quality, ensuring that data remains accurate and reliable from creation to disposal. * **Conformity:** Conformity measures the degree to which data conforms to standardized data definitions. It ensures that data adheres to predefined formats, structures, and naming conventions, facilitating consistency and interoperability across different systems and processes. #### User Defined Dimensions Users also have the option to define custom dimensions, which can be created within the Data Quality -> System Settings.
#### Scheduling Execution The Schedule option allows users to configure automated execution of Data Quality Rules at predefined intervals. Scheduling helps ensure continuous monitoring of data quality without requiring manual execution. **Users can define:** * Execution date and time * Recurrence frequency * Execution intervals based on business requirements Once configured, the rule executes automatically according to the defined schedule and evaluates the associated data objects using the configured validation criteria. Users can create, modify, or remove schedules at any time to accommodate changing business and operational requirements. Automated scheduling enables organizations to proactively monitor data quality, identify issues early, and ensure ongoing compliance with data governance standards. {% hint style="info" %} Users with the appropriate permissions can configure and manage schedules for Data Quality Rules irrespective of the rule status. This provides flexibility to adjust execution timing and frequency as business requirements evolve. {% endhint %}
#### Configuring Notifications The Notifications section allows users to configure how Data Quality Rule execution results are communicated and how the system responds when a rule generates a Success, Warn, Failure, or Execution Failure result. These settings help users proactively monitor data quality issues, automate remediation workflows, and improve governance of data assets.
The following notification and response options are available: **Send Alert On Failure** Enabling Send Alert On Failure sends notifications when the Data Quality Rule execution result is classified as Failure. Notifications are delivered through the channels configured in the user's profile, which may include: * Inbox notifications * Email notifications * Slack notifications * Google Chat notifications Users can manage their notification preferences from My Profile. Rule execution notifications can be viewed from: **My Profile > Inbox > Data Quality**
**Include Detailed Report in Notifications** This option controls the level of information included in Data Quality Rule notifications. **Disabled (Default)** Notifications include only a summary of the rule execution results. **Enabled** Notifications include: * Execution summary * Object-level results * Failed value details Detailed reports are attached to supported notification channels, enabling users to perform deeper analysis without navigating to the rule. **Send Alert On Execution Failure** Enabling Send Alert On Execution Failure sends notifications when the rule execution cannot be completed due to operational or system-related issues. **Examples include:** * Connector availability issues * Query execution errors * Network interruptions * System failures * Permission-related issues This notification helps users quickly identify and resolve execution problems that prevent the rule from running successfully. **Create Service Request On Failure** Enabling Create Service Request On Failure automatically generates Service Requests when associated objects fail Data Quality Rule validation. Service Requests provide a structured workflow for tracking, assigning, and resolving data quality issues. **Generated requests can be viewed from:** Data Quality Rule Summary > Service Requests Service Requests also contribute to Data Quality Score calculations for associated objects. **Do Not Report Duplicate Service Requests** This option is available only when Create Service Request On Failure is enabled. When enabled, the system prevents duplicate Service Requests from being created for the same issue. A new Service Request is created only when no existing request is in an active state. The following statuses are considered closed and do not prevent the creation of a new Service Request: * Closed * Resolved * Rejected * Expired * Deleted This helps reduce duplicate remediation efforts and prevents unnecessary ticket creation. **Apply Caution to Downstream Objects Upon Failure** Enabling this option automatically marks downstream objects as Caution when the Data Quality Rule fails. This behavior helps users identify data assets that may be affected by upstream data quality issues and supports impact analysis across the data lineage. **Display Failed Values in Tables Containing Any Masked/Restricted Column** This option is available only for Code-based Data Quality Rules. When enabled, failed values can be displayed even when the associated tables contain masked or restricted columns, subject to applicable security and governance controls. When disabled, failed values associated with restricted data remain hidden to protect sensitive information. {% hint style="info" %} Notifications generated by Data Quality Rules are delivered to the assigned Steward and configured recipients through the notification channels enabled in their user profile, such as Inbox, Email, Slack, and Google Chat. {% endhint %} #### Updating Steward Each Data Quality Rule is assigned a Steward who is responsible for monitoring rule execution, reviewing failures, and coordinating remediation activities. The Steward helps ensure that the rule continues to meet business and data quality requirements by: * Reviewing rule execution results * Monitoring alerts and notifications * Managing remediation activities * Coordinating with stakeholders to resolve data quality issues * Maintaining rule configurations when business requirements change By default, the rule creator is assigned as the Steward. Users with appropriate permissions can update the Steward and assign ownership to another qualified user.
## Use Case A Customer Relationship Management (CRM) system stores customer contact information that is used for marketing campaigns, customer support, and business communications. Invalid email addresses can result in failed communications, reduced campaign effectiveness, and inaccurate customer data. To ensure email addresses comply with the organization's data quality standards, users can create a Data Quality Rule that validates email values against a predefined email format. ### Solution In this scenario, users can configure the Data Quality Rule with the following settings:
ConfigurationValue
Object TypeTable Columns
FunctionValid Email Format
Success CriteriaResult Value Percent Greater Than or Equal To 100
Warn CriteriaResult Value Percent Greater Than or Equal To 95
Associated ObjectsCustomer Email Address columns
Rule NameEmail Validation
PurposeValidate customer email addresses to ensure they conform to the expected email format and support reliable business communications.
DimensionValidity
Alert On WarnEnabled
Send Alert On FailureEnabled
Add Failed Values to Remediation CenterEnabled
#### Expected Outcome During execution, the system evaluates each email address in the associated columns against the configured email format validation logic. * If 100% of the email addresses satisfy the validation criteria, the rule result is classified as Success. * If the percentage of valid email addresses falls within the configured Warn threshold but does not meet the Success Criteria, the rule result is classified as Warn. * If the percentage of valid email addresses does not satisfy either the Success Criteria or Warn Criteria, the rule result is classified as Failure. After execution, the rule provides detailed execution statistics, including: * Total Rows Evaluated * Passed Row Count * Failed Row Count * Result Value * Execution Result (Success, Warn, or Failure) * Sample Failed Values * Failed Value Details (when available) If remediation is enabled, failed values can be sent to the Data Quality Remediation Center (DQRC) for further investigation and remediation. Notifications are sent through the configured notification channels based on the rule outcome, and Service Requests are created only when the rule result is classified as Failure and the Create Service Request On Failure option is enabled. This configuration helps organizations proactively monitor email quality, identify data quality issues before they become critical, and maintain accurate customer contact information for business operations. ## Accessing Data Quality Rules The Data Quality Rules landing page displays a list of all current rules. Users can click on any rule name to access the specific rule, which redirects them to the summary tab for that particular data quality rule. ### Dashboard The Rule Execution Dashboard provides a quick view of how a data quality rule performed in its most recent run and over the last few runs. It displays both object-level and row-level results, along with key metrics such as execution time, steward, and service requests.
**Top KPIs (Cards)** * **Objects Evaluated:** Number of objects (such as tables, columns, or files) that were checked in the last execution. * **Objects Passed:** Number of objects that passed the rule in the last execution. Clicking on this card opens the Object Execution Results page filtered to show only passed objects. * **Objects Failed:** Number of objects that failed the rule in the last execution. Clicking on this card opens the Object Execution Results page filtered to show only failed objects. * **Rows Evaluated:** Total number of rows that were checked in the last execution. **Last Execution Summary** * **Execution Result:** Shows whether the last execution passed, failed, or had another outcome. * **Execution ID:** A unique ID for the last execution. Clicking on this ID opens the Rule Executions page filtered by this execution. * **Rows Sent for Remediation:** Number of rows from this execution that were sent to the Remediation Center. Clicking on this value opens the Remediation Center filtered to show these rows. * **Execution Time:** The date and time when the last execution ran. * **Rule Steward:** The steward or owner responsible for this rule. * **Service Requests Raised:** Number of service requests that were created during the last execution. Clicking on this value opens the Service Requests page filtered for this rule. **Charts** * **Object Results:** A breakdown of the last execution results at the object level. Results are grouped into: Passed, Failed, Undetermined, and Execution Failed. * **Object Quality Trend (Last 10 Executions):** Shows how object results changed over the last 10 runs. (X-axis: Execution IDs) (Y-axis: Percentage of passed objects (calculated as passed objects ÷ total objects × 100)) * **Row Results:** A breakdown of the last execution results at the row level, showing passed and failed counts. * **Row Quality Trend (Last 10 Executions):** Shows how row results changed over the last 10 runs. (X-axis: Execution IDs) (Y-axis: Percentage of passed rows (calculated as passed rows ÷ total rows × 100)) ### Summary Tab The Summary tab provides a centralized workspace for defining, configuring, and reviewing a Data Quality Rule (DQR). It allows users to set the rule purpose, function, inputs, success criteria, dimension, remediation behavior, notifications, scheduling, and metadata. The page also displays execution history and audit information, supporting transparency and governance.
**Execution Information** * Last Status: Displays the most recent execution status of the rule. Possible values include: * Not Executed * Undetermined * Passed * Execution Failed * Failed * Last Run Time: Shows the date and time at which the rule was last executed. **Purpose:** Displays the purpose statement of the rule. This description should be clear and concise, explaining what the rule validates and why it is important. **Function:** The Function section contains configuration details for the Data Quality Function applied to the rule. Users can define or modify function settings only if the rule has not been executed at least once. {% hint style="info" %} If the rule has never been executed, users may update the function or object type. However, changing either of these will remove all previously associated objects from the rule. {% endhint %} **Function Name:** Displays the name of the Data Quality Function used in the rule (e.g., File Format Validation). * **Input:** Displays the condition, value, pattern, or threshold used by the function during rule execution. Depending on the selected function, the input may consist of numeric values, text values, date values, comma-separated values, or patterns such as regular expressions. * **Success Criteria:** Defines the condition that must be satisfied for the rule execution result to be classified as Success. Users can configure count-based or percentage-based thresholds using supported operators such as Equal To, Greater Than, Less Than, and In Between. During execution, the system compares the calculated result against the configured Success Criteria. * **Warn Criteria:** Defines the condition that must be satisfied for the rule execution result to be classified as Warn. Warn Criteria provides an intermediate severity level between Success and Failure, allowing users to identify potential data quality issues before they become critical failures. When configured, the system evaluates the rule using the following logic: * **Success** – Result satisfies the Success Criteria. * **Warn** – Result satisfies the Warn Criteria but does not satisfy the Success Criteria. * **Failure** – Result satisfies neither the Success Criteria nor the Warn Criteria. Warn Criteria is supported for User Defined Functions that have Support Warn Criteria enabled. * **Dimension:** Represents the Data Quality Dimension assessed by the rule. Supported dimensions include: * Completeness * Uniqueness * Accuracy * Conformity * Integrity * Timeliness * Validity * Consistency These dimensions categorize the type of data quality validation performed. **Data Quality Remediation** Stewards and Admins can configure remediation-related metadata even when remediation is not enabled. All metadata fields remain editable regardless of the Add Failed Values to Remediation Center setting. {% hint style="info" %} Remediation Center support is now available for all Data Quality Functions. Any function can send failed values to the Remediation Center when enabled. {% endhint %} * **Monetary Value:** Represents the business impact cost associated with each failed record.\ Helps remediation owners prioritize issues based on severity and potential risk. * **Criticality:** Indicates the rule’s severity level (Low, Medium, High). Used to prioritize remediation efforts and resource allocation. * **Violation Message:** Provides a descriptive explanation of the rule violation when the execution results in failure. * **Corrective Action:** Describes the recommended steps for resolving the issue.\ Users can also tag a Data Story to add business context or collaborate on remediation planning. * **Avoid Duplicate Failures:** Controls whether duplicate failed values should be reported to the Remediation Center. * Yes: Only unique failures are logged * No (default): All failures are logged * **Add Failed Values to Remediation Center (Toggle):** Determines whether failed records are sent to the Data Quality Remediation Center (DQRC). **Tags** The Tags section displays the tags associated with the Data Quality Rule. Tags help categorize and organize rules, making them easier to search, filter, and manage across the application. Users can add, update, or remove tags to align the rule with business domains, data governance initiatives, projects, or organizational standards. Selecting a tag opens its details and allows users to view additional information about the associated tag. Proper use of tags improves discoverability and helps users quickly identify related Data Quality Rules. **Additional Fields** The Additional Fields section displays custom metadata fields configured by the organization to capture information beyond the standard rule attributes. These fields can be configured to store various types of information, including: * Text values * Numeric values * Date values * Code values * Dropdown selections Organizations can use Additional Fields to capture business-specific, regulatory, operational, or governance-related information relevant to the Data Quality Rule. Users with the appropriate permissions can view and update these fields, enabling organizations to maintain additional context and metadata required for reporting, governance, and compliance purposes. For custom Text fields, users can create rich and structured content using the editor available within the field. The editor allows users to: * Format text using styling options such as bold, italic, and underline * Create ordered and unordered lists * Insert hyperlinks * Add images * Embed videos These capabilities enable users to document detailed business context, implementation notes, governance requirements, and other supporting information directly within the Data Quality Rule. **Top Users** Displays users who engaged most frequently with the rule. * The Steward appears at the top by default. * Other users are listed based on activity count. Each visit to the Summary page increases a user’s score by one. **Notifications and Alert Settings** The Notifications and Alert Settings section allows users to configure how Data Quality Rule execution results are communicated and how the system responds to Warn, Failure, and Execution Failure events. **Send Alert On Failure:** Sends notifications when the rule execution result is classified as Failure. Notifications are delivered through the channels configured in the user's profile, such as Inbox, Email, Slack, or Google Chat. * Include Detailed Report in Notifications: Controls the amount of information included in rule execution notifications. * Disabled (Default): Notifications include only a summary of the execution results. * Enabled: Notifications include both the execution summary and detailed failed-value information, where applicable. **Send Alert On Execution Failure:** Sends notifications when the rule execution cannot be completed due to operational or system-related issues, such as: * Connector unavailability * SQL/query errors * System failures * Network disruptions **Send Alert On Warning:** Sends notifications when the rule execution result is classified as Warn. Warn notifications help users proactively identify and address potential data quality issues before they escalate into failures. Notifications are delivered through the channels configured in the user's profile. **Create Service Request On Failure:** Automatically creates a Service Request when the rule execution result is classified as Failure. Generated Service Requests can be tracked and managed from the Service Requests tab of the Data Quality Rule. **Do Not Report Duplicate Service Requests:** Prevents duplicate Service Requests from being created for the same issue. When enabled, a new Service Request is created only if there is no existing active Service Request for the affected object. **Apply Caution to Downstream Objects Upon Failure:** Automatically marks downstream objects as Caution when the rule execution result is classified as Failure. This helps users identify data assets that may be impacted by upstream data quality issues. **Code Rule–Specific Option:** Display Failed Values in Tables Containing Any Masked/Restricted Column Available only for Code-based Data Quality Rules. When enabled, failed values can be displayed for tables containing masked or restricted columns, subject to applicable security and governance policies. When disabled, failed values associated with restricted data remain hidden. **Schedule** Allows users to schedule the rule to run automatically.\ Users can: * Choose execution dates * Set execution times * Configure recurrence intervals If no schedule is defined, the system displays:\ No scheduled jobs have been found. **Audit History** Audit History shows a complete log of all rule updates, including: * Status changes (Draft ↔ Active) * Function and dimension updates * Changes to associated objects * Updates to remediation settings * Schedule changes Each entry includes: * Modified fields * Previous and updated values * User who made the change * Timestamp This ensures full traceability and governance. ### Associated Objects The Associated Objects tab displays all objects linked to a Data Quality Rule. Depending on the function and rule configuration, the associated objects may include Tables, Table Columns, Files, File Columns, or Code. This tab helps users view contextual information, configure object-level settings, review the latest execution results, and adjust rule execution behavior for each individual object. All fields in this tab dynamically update based on the type of object associated with the rule. #### **Table**
When a Data Quality Rule includes Table objects, the Associated Objects tab displays the following columns. * **Database:** Displays the database to which the table belongs. Users can apply filters to narrow down tables across multiple databases. * **Schema:** Shows the schema under which the table resides. Filtering this column helps users focus on specific schemas. * **Table:** Displays the name of the table associated with the rule. This helps users quickly identify which table is included in the rule configuration. * **Object Status:** Indicates whether the table is Active or Inactive within the rule. Inactive tables are excluded from rule execution. * **Last Run Date:** Displays the timestamp when the Data Quality Rule was last executed for this table. Users can sort this column to identify the most recent or oldest execution. * **Last Object Execution Result:** Shows the most recent execution result for the table—Passed, Failed, Execution Failed, or Undetermined.
A quick-view eye icon provides a summary including: * **Result Value:** Displays the value calculated for the object during rule execution. This may be a count or percentage depending on how the rule’s success criteria are configured. * **Input:** Shows the input parameter or condition applied to the rule function for the selected object. This field reflects the value used during execution when object-level inputs are configured. * **Success Criteria:** Indicates the condition used to determine whether the selected object has passed or failed the rule evaluation (for example, “Result Value must be greater than 95%”). * **Maximum Failed Values for Remediation:** Displays the maximum number of failed values that can be sent to the Remediation Center for this object, based on the configuration set at the rule or system level. * **Service Request ID:** Shows the Service Request generated for the selected object during rule execution, when the Create Service Request on Failure setting is enabled. * **Failed Values sent to the Remediation Center:** Indicates the number of failed values forwarded to the Data Quality Remediation Center for this object during execution. * **Downstream Objects Cautioned:** Displays the number of downstream objects that were marked with caution due to the object’s failed rule result, based on the Apply Caution to Downstream Objects setting. * **History:** Clicking History directs users to the Object Execution Results tab, which displays complete execution details for the selected table, including result values, row statistics, and failed values. * **Advanced Settings:** For Table objects, the Advanced Settings section provides the Function Inputs option, which is divided into Default and Custom views. ![](/files/Q0SYEKbyX3vm3oiCLGZQ) * **Default:** Displays the Success Criteria configured at the rule level. These settings are read-only and reflect the default validation logic applied during execution. * **Custom:** Allows users to override the default configuration for the selected table. Users can edit the Success Criteria and Add or modify the Violation Message * **Success Criteria:** Success Criteria define the acceptance condition for validating the rule execution.\ If the result value for the table falls within the specified range or matches the configured value, the rule is marked Passed; otherwise, it is marked Failed. * Success Criteria support two validation modes: Count and Percentage * These modes allow users to evaluate the result based on actual row counts or their percentage distribution. * **Example:** If the rule requires 95% or higher accuracy, users can select: * Validation Mode: % * Operator: In Between * Values: 95 and 100 * **Violation Message:** The Violation Message explains why the rule failed for the selected table, based on the rule’s dimension (e.g., Completeness, Accuracy, Consistency). Users can define a custom violation message under the Custom view. * When configured, this message appears: * In the Data Quality Remediation Center (DQRC) * Within execution summaries * As part of the failure explanation provided to end-users * **Add Table:** The Add Table option enables users to include additional table objects in a Data Quality Rule. Users can select multiple tables using search filters such as Connector Name, Schema, and Table. Once added, each table appears in the Associated Objects tab, where its object-level settings can be configured. #### **Table Columns**
When a Data Quality Rule includes Table Column objects, the Associated Objects tab displays the following columns. These fields provide contextual information, object-level configuration options, and execution details specific to each table column. * **Database:** Displays the database to which the table column belongs. Users can filter this field to locate columns across multiple databases. * **Schema:** Shows the schema under which the table column resides. Filtering helps users focus on specific schemas. * **Table:** Displays the name of the table that contains the column. This helps users understand the structural context of the column. * **Column:** Shows the name of the column included in the Data Quality Rule. This is the primary object being validated during execution. * **Max Failed Values Limit:** Displays the object-level configuration option for setting the maximum number of failed values to be reported to the Data Quality Remediation Center. The maximum limit is controlled through the system setting: dataquality.remediationcenter.failedrecords.max.limit. * **Object Status:** Indicates whether the table column is Active or Inactive within the rule. Inactive columns are excluded from rule execution. * **Last Run Date:** Displays the timestamp when the rule was last executed for this column. Users can sort the column to view the most recent or earliest execution. * **Last Object Execution Result:** Shows the most recent execution result for the column—Passed, Failed, Execution Failed, or Undetermined. * An eye icon provides a detailed summary of the last execution, including Result Value, Input, Success Criteria, Failed Values sent to the Remediation Center, and more.
* **History:** Clicking History navigates to the Object Execution Results tab, which displays detailed execution information for the selected table column. * **Primary Columns:** Primary Columns allow users to display one or more key columns along with the failed value during rule execution. This enhances identification of failed records by showing the failed value alongside the selected primary columns, making it easier to determine the specific row that caused the issue. * **Additional Columns:** Additional Columns provide users the option to display extra contextual columns along with the failed value. This helps enrich the information shown for failed rows, enabling clearer identification and deeper understanding of the problematic data. * **Advanced Settings:** For Table Column objects, the Advanced Settings section includes two tabs: Filter and Function Inputs. These options allow users to configure column-level execution behavior, apply filters, adjust function inputs, and manage object-level query settings.
* **Filter:** The Filter tab provides options that allow users to refine and control how data is selected for rule execution at the column level. * **Parameterized Filter (WHERE Clause):** Allows users to add WHERE clause conditions to filter the dataset used during rule execution. This ensures the rule runs only on rows that meet the specified conditions. * **Incremental Data Filtering:** Enables filtering of data starting from the last execution timestamp, ensuring that the rule validates only new or updated rows instead of scanning all records. This option is available only if the table contains one or more date or timestamp-type columns. If no eligible date column exists, the option appears disabled. * **Rule Query:** Displays the Data Quality Function query generated for the table column. This query can be reviewed and modified only when the rule is in Draft state. * **Statistics Query:** Shows the query used to calculate statistics such as total count, passed count, and failed count (depending on the function). This query is also editable only in Draft state. * **Failed Value Query:** Shows the query that extracts failed values for this table column. Users can modify this query if needed, but only when the rule is in Draft state. * **Function Inputs:** The Function Inputs tab contains two views: * **Default:** Displays the Input and Success Criteria configured at the rule level. These fields are read-only and show the default validation logic. * **Custom:** Allows users to override the rule-level configuration for this specific table column. * **Users can modify:** * **Input:** Column-level function input parameters. * **Success Criteria:** Column-level acceptance conditions that determine whether the object passes or fails. * These column-specific settings allow the same rule to behave differently for different table columns when required. * **Add Table Column:** The Add Table Column option enables users to include additional table column objects in a Data Quality Rule. Users can search and select columns using filters such as Connector Name, Schema, Table, and Column. Once added, each column appears in the Associated Objects tab, where its object-level settings, filters, and function inputs can be configured. #### **Files**
When a Data Quality Rule includes File objects, the Associated Objects tab displays the following columns. These fields provide contextual information, execution details, and configuration options for each file associated with the rule. * **File System:** Displays the connector associated with the file. This identifies the source system (e.g., S3, ADLS, GCS) from which the file is accessed. * **Profile Score:** Displays the profiling method used for the file, such as Direct or DuckDB, indicating how the file was profiled within the system. This helps users understand the profiling approach applied to the file rather than a numerical data quality score. * **File Name:** Displays the name of the file included in the rule. This is the primary object being validated during rule execution. * **File Location:** Shows the directory or folder path where the file resides. Users can filter this column to locate files within specific directories or paths. * **Object Status:** Indicates whether the file is Active or Inactive within the rule. Inactive files are excluded from rule execution. * **Last Run Date:** Displays the timestamp when the Data Quality Rule was last executed for this file. Users can sort this column to review the latest or earliest executions. * **Last Object Execution Result:** Shows the most recent execution result for the column—Passed, Failed, Execution Failed, or Undetermined. * An eye icon provides a detailed summary of the last execution, including Result Value, Input, Success Criteria, Failed Values sent to the Remediation Center, and more.
* **History:** Clicking History navigates to the Object Execution Results tab, which displays detailed execution information for the selected table column. * **Advanced Settings:** For File objects, the Advanced Settings section contains the Function Inputs option only. Files do not support Filter or Query modification settings, making configuration simpler compared to Table and Table Column objects. ![](/files/DQMJr0JSsoxLPWPdG8Y3) * The Function Inputs tab includes two views: * **Default:** Displays the Input and Success Criteria configured at the rule level. These fields are read-only and reflect the default validation logic. * **Custom:** Allows users to override the rule-level configuration for this specific file. * **Users can modify:** * **Input** – File-level function input parameters * **Success Criteria** – Conditions that determine whether the file passes or fails Object-level configuration allows the rule to behave differently for different files when required. * **Add File:** The Add File option enables users to include additional file objects in a Data Quality Rule. * Users can search for files using filters such as: * Connector Name * File Name * File Location / Directory Path Once selected, each file appears in the Associated Objects tab, where users can configure its object-level settings and function inputs. #### **File Columns** When a Data Quality Rule includes File Column objects, the Associated Objects tab displays the following columns. These fields provide contextual information, object-level configuration options, and execution details specific to each file column.
* **File System:** Displays the connector associated with the file. This identifies the source system (e.g., S3, ADLS, GCS) from which the file is accessed. * **Profile Source:** Displays the profiling method used for the file, such as Direct or DuckDB, indicating how the file or file column was profiled within the system. * **File Name:** Shows the name of the file that contains the column. This helps users identify the context of the file column being validated. * **File Column Name:** Displays the name of the column within the file. This is the primary object evaluated during rule execution. * **Max Failed Values Limit:** Displays the object-level configuration setting that determines the maximum number of failed values that can be reported to the Data Quality Remediation Center. This value is controlled through the system setting: ***dataquality.remediationcenter.failedrecords.max.limit***. * **File Location:** Shows the directory or path where the file resides. Users can filter this column to narrow results and locate specific files. * **Object Status:** Indicates whether the file column is Active or Inactive within the rule. Inactive columns are excluded from rule execution. * **Last Run Date:** Displays the timestamp when the Data Quality Rule was last executed for this file column. Users can sort this column to view the most recent or earliest run. * **Last Object Execution Result:** Shows the most recent execution outcome—Passed, Failed, Execution Failed, or Undetermined. * An eye icon provides a detailed summary of the last execution, including Result Value, Input, Success Criteria, Failed Values sent to the Remediation Center, and more.
* **History:** Clicking History navigates to the Object Execution Results tab, which provides complete execution details for the selected file column. This includes statistics, row-level outcomes (if supported by the function), and any identified failed values. * **Advanced Settings :** For File Column objects, the Advanced Settings section works similarly to Table Column, providing two key configuration tabs: Filter and Function Inputs.
* **Filter** * **Parameterized Filter (WHERE Clause):** Allows users to append WHERE clause conditions to filter the dataset used for rule execution. This ensures that the rule validates only rows that match the specified conditions. * **Incremental Data Filtering:** Evaluates only new or updated records by filtering data starting from the last execution timestamp. This option is enabled only if the file contains one or more columns with a date or timestamp data type. If no such column is present, this option appears disabled. * **Rule Query:** Displays the function-level query generated for the file column. Users may review and modify this query only when the rule is in Draft state. * **Statistics Query:** Shows the query used to compute statistics such as total row count, passed row count, or failed row count (depending on the function). This query can also be edited only in Draft state. * **Failed Value Query:** Displays the query used to extract failed values for this file column.\ It can be edited as needed, but only when the rule is in Draft state. * **Function Inputs:** The Function Inputs tab provides two configuration views: * **Default:** Displays the Input and Success Criteria predefined at the rule level. These values are read-only and represent the default validation logic. * **Custom:** Allows users to override the rule-level configuration for this specific file column. * Users can modify: * **Input** – Column-level input parameters * **Success Criteria** – Conditions that determine whether the file column passes or fails * These object-specific configurations allow the rule to behave differently for different file columns when required. * **Add File Column:** The Add File Column option enables users to include additional file column objects in a Data Quality Rule. Users can search and select columns using filters such as: * File System (Connector Name) * File Name * File Location * File Column Name Once added, each file column appears in the Associated Objects tab, where users can configure its object-level settings, filters, and function inputs. #### **Code** When a Data Quality Rule includes Code objects, the Associated Objects tab displays fields that describe the SQL-based validation logic, associated objects extracted from the rule query, and recent execution details. Code objects rely entirely on SQL statements defined by the user rather than physical table or file selections.
Fields Displayed for Code Objects * **Database:** Displays the database against which the Code object is executed. This helps users understand the execution context of the SQL provided in the Rule Query. * **Rule Query:** Displays the SQL query that defines the validation logic for the Code object. This query is written by the user and determines how data is evaluated for quality. * **Result State Query:** Displays the query used to determine the success or failure of the rule. The result of this query is compared against the defined Success Criteria to generate the final rule result. * **Failed Data Query:** Displays the SQL query that extracts the failed records when the rule execution results in failure. These failed records may be forwarded to the Data Quality Remediation Center, depending on system configuration. * **Object Status:** Indicates whether the Code object is Active or Inactive within the rule. Inactive objects are excluded from rule execution. * **Last Run Date:** Displays the timestamp when the Code object was last executed. Users can sort this field to view the most recent or earliest execution. * **Last Object Execution Result:** Shows the most recent execution outcome—Passed, Failed, Execution Failed, or Undetermined. An eye icon provides a detailed summary of the last execution, including Result Value, Input, Success Criteria, Failed Values sent to the Remediation Center, and more. * **Additional Associations:** When the rule is in Draft state, the + icon appears next to the Code object. Clicking this opens the Add Associate Object interface. ![](/files/erLzWZItgrirQJK69YYo) * This section lists all objects referenced in the SQL queries—either detected automatically or added manually—and displays the following columns: * **Database:** The database referenced in the SQL query. * **Schema:** The schema extracted from the referenced object. * **Table:** The table referenced within the SQL query. * **Column:** The specific column referenced within the SQL query. * **Added By:** Shows the user who added the associated object, either through auto-population or manual selection. * **Added On:** Displays the timestamp when the object association was created. * This view helps users verify, audit, and manage all associated objects referenced in the Code logic. * **History:** Selecting History navigates to the Object Execution Results tab. This displays detailed results for the Code object, including result values, computed statistics, and any failed records (if supported by the rule configuration). ## **Additional Information** When a user provides a Rule Query while creating or editing a Data Quality Rule, the system automatically analyzes the SQL statement to identify all referenced data objects. This automated discovery ensures that all dependent objects are correctly associated with the rule. ### System Behavior The system uses the Gudu SQL Parser to extract metadata components from the Rule Query, including: Database, Schema, Tables and Columns. If the parser identifies these components successfully: * The associated Database, Schema, Table, and Column objects are automatically populated in the Associated Objects section. * These appear in the Field-Level Query Parameters view, providing visibility into all referenced objects. In some cases, the parser may be unable to detect all objects due to: * Complex SQL * Nested or dynamic queries * Unsupported syntax To ensure complete execution accuracy, the system provides a manual override option: #### Add Associate Object (+) Users can manually add any missing objects referenced in the SQL query by selecting them from the metadata catalog. Users are able to choose: Database, Schema, Table and Column. This guarantees that all necessary objects are correctly associated with the rule. Auto-Populated Associated Objects: Objects identified by the parser appear automatically based on the SQL provided in the Rule Query. Manual Add Option (+ Add Associate Object): Enables users to associate objects manually when the parser fails to detect them. ### **Object Status** When a connector license is removed, the system automatically updates the status of all related objects linked to active Data Quality Rules to Inactive. A background job or system trigger performs this update and may log the reason, such as “Connector license removed.” These inactive objects remain visible on the rule configuration page with clear status indicators and tooltips to help users understand their current state. For custom functions, the system also validates whether the required data types are present. If the expected data types are not supported, the object is marked Inactive to prevent invalid rule execution. ### **Nine Dots Options (Associated Objects)** The Nine Dots menu provides several actions that users can perform within a Data Quality Rule. However, on the Associated Objects tab, only a limited set of options is available. The following actions appear in the menu, but some are disabled based on the nature of the References tab. * **Change Rule Status:** Allows users to change the Data Quality Rule status between Active and Draft. * **Run Now:** Executes the Data Quality Rule. The rule must be in the Active state. * **Activate Objects:** Marks selected associated objects as Active so they are included in rule execution. * **Inactivate Objects:** Marks selected associated objects as Inactive, excluding them from rule execution. * **Update Steward:** Allows users to update the steward of the Data Quality Rule. Available only when the rule is in the Draft state. * **Regenerate Object Queries:** Regenerates Data Quality Function object queries when updates are made to the Functions Template. A background job performs the regeneration. This option is enabled only for objects of type Code. * **Remove Associated Objects:** Removes selected associated objects from the Data Quality Rule. Available only when the rule is in the Draft state. * **Delete Contacts:** Removes added contacts from the Data Quality Rule. Available only when the rule is in the Draft state. * **Delete Rule:** Deletes the Data Quality Rule. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.ovaledge.com/release8.2/data-quality/data-quality-rules/creating-data-quality-rule.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.