Schema Registry
Schema Registry provides a centralized repository for managing and validating schemas for topic message data, as well as serializing and deserializing it over the network. As schemas evolve, producers and consumers of Kafka topics can use schemas to ensure data consistency and compatibility.
OvalEdge, on the other hand, offers a user-friendly interface that enables connectivity to Schema Registry and supports a wide range of standard operations. It integrates with Schema Registry using the JDBC driver, which allows for tasks such as crawling database objects, profiling sample data, executing queries, building lineage, implementing remote policies, and managing access permissions.
Prerequisites
The following are prerequisites for connecting to the Schema Registry:
The APIs/ drivers used by the connector are given below:
Sl.No
Driver / API
Version
Details
1
API
User Permission
An admin/service account for crawling and building lineage. The minimum privileges required are:
Operation
Access Permission
Connection validate
Read
Crawl datasets
Read
Configuring Environment Variables (Optional)
Configuring environment names enables you to select the appropriate environment from the drop-down list when adding a connector. This allows for consistent crawling of schemas across different environments, such as production (PROD), staging (STG), or temporary environments. It also facilitates schema comparisons and assists in application upgrades by providing a temporary environment that can be later deleted if needed.
Before establishing a connection, it is important to configure the environment names for the specific connector. If your environments have been configured, skip this step.
Go to Administration > System Settings.
Select the Connector tab and find the Key name "connector.environment".
Enter the desired environment values (e.g., PROD, STG) in the value column.
Click on the checkmark ✔ icon to save the configuration.
Technical Specification
The connector capabilities are shown below:
Crawling
Feature
Supported Objects
Remarks
Crawling
Tables
Table Columns
Supported Data Types:
Bigint, Char, Date, Datetime, Datetime2, Decimal (P, S), Float, Int, Money, Nchar, Ntext, Numeric (P, S), Nvarchar, Real, Smalldatetime, Smallint, Smallmoney, Time, Tinyint, Varchar
Views
Stored Procedures
-
Functions
-
Profiling
Feature
Supported Objects
Remarks
Table Profiling
Row Count, Columns Count, View Sample data
-
View Profiling
Row Count, Columns Count, View sample data
View is treated as a table for profiling purposes.
Column Profiling
Min, Max, Null Count, Distinct, Top 50 values
-
Full Profiling
Supported
-
Sample Profiling
Supported
-
Lineage Building
Lineage entities
Details
Table Lineage
Supported
Column Lineage
Supported
Lineage Sources
Stored Procedures, functions, triggers, views, SQL queries (from Query Sheet)
Querying
Operation
Details
Select
Supported
Insert
Not supported, by default.
Update
Not supported, by default.
Delete
Not supported, by default.
Joins within database
Supported
Joins outside database
Not supported
Aggregations
Supported
Group By
Supported
Order By
Supported
By default, the service account provided for the connector will be used for any query operations. If the service account has to write privileges, then Insert / Update / Delete queries can be executed.
Connection Details
To connect to the Schema Registry using the OvalEdge application, complete the following steps:
Log in to the OvalEdge application.
Navigate to Administration > Connectors.
Click on the + (New Connector) icon, and the Add Connection with Search Connector pop-up window is displayed.
Add Connector pop-up window is displayed where you can search for the Schema Registry connector.
The Add Connector with Connector Type specific details pop-up window is displayed. Enter the relevant information to configure the Schema Registry connection.
Note: The asterisk (*) denotes mandatory fields required for establishing a connection.
Field Name
Description
Connector Type
It allows you to select the connector from the drop-down list. By default, 'Schema Registry' is displayed as the selected connector type.
Connector Settings
License Add Ons
All the connectors will have a Base Connector License by default that allows you to crawl and profile to obtain the metadata and statistical information from a datasource.
OvalEdge supports various License Add-Ons based on the connector’s functionality requirements.
Auto Lineage: Select the Auto Lineage Add-On license that enables the automatic construction of the Lineage of data objects for a connector with the Lineage feature.
Data Quality: Select the Data Quality Add-On license to identify, report, and resolve the data quality issues for a connector whose data supports data quality, using DQ Rules/functions, Anomaly detection, Reports, and more.
Data Access: Select the Data Access Add-On license that will enforce connector access via OvalEdge with Remote Data Access Management (RDAM) feature-enabled.
Connector Name*
Provide a connector name for the Schema Registry database in OvalEdge. This name will serve as a reference to identify the specific Schema Registry database connection. For example, you can use "SchemaRegistry_Connection_test" as the connector name.
Connector Environment
The Connector Environment drop-down list allows you to select the environment configured for the connector from the drop-down list.
For example, PROD, or STG (based on the configured items in the OvalEdge configuration for the connector.environment).
The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA).
Note: The steps to set up environment variables are explained in the prerequisite section.
Schema Registry Endpoint*
Cloud API Key*
Cloud API Secret*
Default Governance Roles
Steward*
Select the Steward from the drop-down options.
Custodian*
Select the Custodian from the drop-down options.
Owner*
Select the Owner from the drop-down options.
Governance Roles 4, 5, 6*
Select the respective user from the drop-down options.
Note: The drop-down list displays all the configurable roles (single user or a team) as per the configurations made in the OvalEdge Security > Governance Roles section.
Admin Roles
Integration Admins*
To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, and then click on the Apply button. The responsibility of the Integration Admin includes configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects.
Security and Governance Admins*
To add Security and Governance Admin roles, search for or select one or more roles from the list, and then click on the Apply button. The security and Governance Admin is responsible for:
Configure role permissions for the connector and its associated data objects.
Add admins to set permissions for roles on the connector and its associated data objects.
Update governance roles.
Create custom fields.
Develop Service Request templates for the connector.
Create Approval workflows for the templates.
Select Bridge
With the OvalEdge Bridge component, any cloud-hosted server can connect with any on-premise or public cloud data sources without modifying firewall rules. A bridge provides real-time control that makes it easy to manage data movement between any source and destination. For more information, refer to Bridge Overview.
For more information, refer to Bridge Overview
After filling in all the connection details, select the appropriate button based on your preferences.
Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
Save & Configure: For certain Connectors that require additional configuration settings. Click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
Once the connection is validated and saved, it will be displayed on the Connectors home page.
Connection Settings
Crawler
Property
Description
Crawler Options
Tables, views & columns
By default, it is selected true
Relationships
Need to select for crawl relationships
Crawler rules
Default includes tables regex has regex = (.*), which can crawl all the tables
Profiler
Sl.No
Property
Description
Remarks
1
Profile Options
Tables and columns
By default, it is selected true
2
Profile Rules
Views and columns
By default, it is selected true
Default includes tables regex has regex = (.*), which can profile all the tables
Profile Rules
Was this helpful?