MySQL

This article outlines the integration with the MySQL connector, enabling streamlined metadata management through features such as crawling, profiling, querying, data preview, and lineage building (both automatic and manual).

MySQL connectivity is established using the JDBC driver. The connector supports User ID & Password Authentication and Azure Entra ID Authentication to establish access to MySQL metadata and data based on the privileges assigned to the service account.

Overview

Connector Details

Connector Category

RDBMS

OvalEdge Release Supported

Release6.3.4 and later

Connectivity

[How the connection is established with RDBMS system]

JDBC driver

Verified MySQL Version

MySQL8.4

The MySQL connector has been validated with the mentioned "Verified MySQL Versions" and is expected to be compatible with other supported MySQL versions. If there are any issues with validation or metadata crawling, please submit a support ticket for investigation and feedback.

Connectivity to the MySQL connector is established using a JDBC driver. The supported driver version is listed below:

Driver Details

Version

MySQL JDBC Driver

8.2.0

Connector Features

Feature

Availability

Crawling

✅

Delta Crawling

❌

Profiling

✅

Query Sheet

✅

Data Preview

✅

Auto Lineage

✅ To learn more, click here.

Manual Lineage

✅

Secure Authentication via Credential Manager

✅

Data Quality

✅

DAM (Data Access Management)

❌

Bridge

✅

Metadata Mapping

The following objects are crawled from MySQL and mapped to the corresponding UI assets.

MySQL Object

MySQL Attribute

OvalEdge Attribute

OvalEdge Category

OvalEdge Type

Table

Table Name

Table

Tables

table

Table

Table Type

Type

Tables

table

Table

Table Comments

Source Description

Descriptions

Source Description

Columns

Column Name

Column

Table Columns

Columns

Data Type

Column Type

Table Columns

Columns

Description

Source Description

Table Columns

Columns

Ordinal Position

Column Position

Table Columns

Columns

Length

Data Type Size

Table Columns

Views

View Name

View

Tables

view

Views

text

View Query

Views

View

Procedures

routine_name

Name

Procedures

Procedure

Procedures

description

Source Description

Descriptions

Procedures

routine_definition

Procedure

Procedures

Functions

routine_name

Name

Functions

routine_definition

Function

Functions

description

Source Description

Descriptions

Triggers

Trigger Name

Name

Triggers

Trigger Definition

Trigger Data

Triggers

Trigger Type

Type

Triggers

Set up a Connection

Prerequisites

The following are the prerequisites to establish a connection:

If Azure Entra ID Authentication is selected as the authentication type, ensure that the application is installed in the Azure environment.

External Supporting Files

The required external JAR files are included as part of the OvalEdge installation artifacts. For driver installation and configuration details, refer to the Connector Drivers Setup Guide. Please contact the OvalEdge Team for assistance related to the driver files and configuration setup.

Whitelisting Ports

Make sure that inbound port “3306” is whitelisted to enable successful connectivity with the MySQL database.

The default port number for the MySQL data source is 3306. If a different port is used, ensure that the updated port number is specified during connection setup, the port is whitelisted, and communication between the system and the MySQL data source is properly established.

Service Account User Permissions

It is recommended to use a dedicated service account to establish the connection to the data source, configured with the following minimum set of permissions.

👨‍💻 Who can provide these permissions? These permissions are typically granted by the MySQL administrator, as users may not have the required access to assign them independently.

Operations

Objects

Sys Tables

Access Permissions

Connection Validation

MySQL Database Connection

Valid database login (Service Account user). No object-level privileges are required. CONNECT/USAGE privilege granted by default to any user is sufficient. If SSH tunneling is enabled, OS-level SSH access and permission to set up local port forwarding are required.

Crawling

Schemas / Tables / Table Columns

INFORMATION_SCHEMA.SCHEMATA / INFORMATION_SCHEMA.TABLES / INFORMATION_SCHEMA.COLUMNS

At least one privilege on the schema, table, or view. Examples include USAGE/SELECT for schemas and SELECT for tables/views.

Crawling

Indexes

INFORMATION_SCHEMA.STATISTICS

A privilege on the target table, such as SELECT/INDEX, so that its index rows are exposed in INFORMATION_SCHEMA.STATISTICS.

Crawling

Column Relations

INFORMATION_SCHEMA.KEY_COLUMN_USAGE

SELECT permission on INFORMATION_SCHEMA.KEY_COLUMN_USAGE and a privilege on the parent/child tables so their constraint rows are visible.

Crawling

Views

INFORMATION_SCHEMA.VIEWS

SHOW VIEW privilege on the schema/view. Without it, VIEW_DEFINITION is returned empty. SELECT privilege on the view and its underlying tables is typically required alongside SHOW VIEW.

Crawling / Lineage

Stored Procedures / Functions

INFORMATION_SCHEMA.ROUTINES

DEFINER / SHOW_ROUTINE

Crawling

Triggers / Full Triggers

INFORMATION_SCHEMA.TRIGGERS

TRIGGER privilege on the schema/table containing the trigger.

Profiling / Sample Profiling / Data Preview / Crawling / Micro DB / Parquet Streaming Read (AskEdgi)

Tables / Table Columns

Target Table / Columns

SELECT permission for the Service Account user on the target table and applicable columns.

Govern Data Query

Tables / Table Columns

Target Table / Columns

SELECT permission for the Service Account user on the target table and columns referenced in the SELECT list and WHERE conditions.

Query Log Crawl

Query Logs

mysql.general_log

general_log must be enabled and log_output must be set to TABLE. Reading the mysql system database typically requires an elevated/administrative grant, such as SELECT on mysql.

Execute Scripts in Transaction

Schemas / Database Objects

Target Schema

Initial checks (select database() / select 1) require only a valid login. Each executed script requires the privileges appropriate to its statements on the target schema.

Profiling / Sample Profiling includes Non-Null Count, First N Rows, Column Profiling (text types / spatial / generic / string / numeric), Top Value Distribution, and Sum / Avg / Std Dev.

Connection Configuration Steps

Users are required to have the Connector Creator role in order to configure a new connection.

Log into OvalEdge, go to Administration > Connectors, click + (New Connector), search for MySQL, and complete the required parameters.

Fields marked with an asterisk (*) are mandatory for establishing a connection.

Field Name

Description

Connector Type

By default, "MySQL" is displayed as the selected connector type.

Authentication

The following two types of authentication are supported for MySQL Server:

UserId & Password Authentication
Azure Entra ID Authentication

Field Name

Description

SSL Enabled*

Select True or False. By default, the value is set to False.

Use SSH Tunnel*

Select True or False. By default, the value is set to False.

SSH Host/IP*

Enter the Host of SSH Tunnel Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH Port*

Enter the SSH Tunnel Port Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH User Name*

Enter the SSH Tunnel User Name Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH Password*

Enter the Password of SSH Tunnel Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

Local Port*

Enter the Local Port for port forwarding Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

Credential Manager*

Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.

Supported Credential Managers:

OE Credential Manager
AWS Secrets Manager
HashiCorp
Azure Key Vault

For more details, click here.

License Add Ons

Select the checkbox for Auto Lineage Add-On to build data lineage automatically.
Select the checkbox for Data Quality Add-On to identify data quality issues using data quality rules and anomaly detection.

For more details, click here.

Connector Name*

Enter a unique name for the MySQL connection

(Example: "MySQLdb").

Connector Environment

Select the environment (Example: PROD, STG) configured for the connector. For more details, click here.

Connector Description

Enter a brief description of the connector.

Server*

Enter the MySQL Server database server name or IP address. (Example: xxxx-sxxxxxx.xxxx4ijxxxl.xx-xx-1.rxs.xxxxx.com or 1xx.xxx.1.xx).

Port*

By default, the port number for MySQL, "3306" is auto-populated. If required, the port number can be modified as per the custom port number that is configured for the MySQL Database.

Database*

Enter the user-defined database name for the MySQL connection to crawl metadata.

Driver*

By default, the MySQL driver details are auto-populated.

Username*

Enter the service account username set up to access the MySQL database (Example: "oesauser").

Password*

Enter the password associated with the service account user.

Connection String

Configure the connection string for the MySQL database:

Automatic Mode: The system generates a connection string based on the provided credentials.
Manual Mode: Enter a valid connection string manually.

Replace placeholders with actual database details.

{sid} refers to the Database Name

Plugin Server

Enter the server name when running as a plugin server.

Plugin Port

Enter the port number on which the plugin is running.

Field Name

Description

SSL Enabled*

Select True or False. By default, the value is set to False.

Use SSH Tunnel*

Select True or False. By default, the value is set to False.

SSH Host/IP*

Enter the Host of the SSH Tunnel. Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH Port*

Enter the SSH Tunnel Port. Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH User Name*

Enter the SSH Tunnel User Name.

Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

SSH Password*

Enter the Password of the SSH Tunnel.

Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

Local Port*

Enter the Local Port for port forwarding.

Note: This field is enabled only when the Use SSH Tunnel option is selected as True.

Credential Manager*

Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.

Supported Credential Managers:

OE Credential Manager
AWS Secrets Manager
HashiCorp
Azure Key Vault

For more details, click here.

License Add Ons

Select the checkbox for Auto Lineage Add-On to build data lineage automatically.
Select the checkbox for Data Quality Add-On to identify data quality issues using data quality rules and anomaly detection.

For more details, click here.

Connector Name*

Enter a unique name for the MySQL connection

(Example: "MySQLdb").

Connector Environment

Select the environment (Example: PROD, STG) configured for the connector. For more details, click here.

Connector Description

Enter a brief description of the connector.

Server*

Enter the MySQL Server database server name or IP address. (Example: xxxx-sxxxxxx.xxxx4ijxxxl.xx-south-1.rxs.xxxxx.com or 1xx.xxx.1.xx).

Port*

By default, the port number for MySQL, "3306" is auto-populated. If required, the port number can be modified as per the custom port number that is configured for the MySQL Database.

Database*

Enter the user-defined database name for the MySQL connection to crawl metadata.

Driver*

By default, the MySQL driver details are auto-populated.

Username*

Enter the service account username set up to access the MySQL database (Example: "oesauser").

Connection String

Configure the connection string for the MySQL database:

Automatic Mode: The system generates a connection string based on the provided credentials.
Manual Mode: Enter a valid connection string manually.

Replace placeholders with actual database details.

{sid} refers to the Database Name AuthenticationPlugins: com.azure.identity.extensions.jdbc.mysql.AzureMysqlAuthenticationPlugin This is part of Azure Identity Extensions for MySQL, which allows connecting to Azure Database for MySQL using Azure AD authentication.

Plugin Server

Enter the server name when running as a plugin server.

Plugin Port

Enter the port number on which the plugin is running.

Default Governance Roles*

Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection.

Admin Roles*

Select one or more users from the dropdown list for Integration Admin and Security & Governance Admin. All users configured in the security settings are available for selection.

No Of Archive Objects*

This shows the number of recent metadata changes to a dataset at the source. By default, it is off. To enable it, toggle the Archive button and specify the number of objects to archive.

Example: Setting it to 4 retrieves the last four changes, displayed in the 'Version' column of the 'Metadata Changes' module.

Select Bridge*

If applicable, select the bridge from the drop-down list.

The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.

After entering all connection details, the following actions can be performed:
1. Click Validate to verify the connection.
2. Click Save to store the connection for future use.
3. Click Save & Configure to apply additional settings before saving.
The saved connection will appear on the Connectors home page.

Manage Connector Operations

Crawl/Profile

To perform crawl and profile operations, users must be assigned the Integration Admin role.

The Crawl/Profile button allows users to select one or more schemas for crawling and profiling.

Navigate to the Connectors page and click Crawl/Profile.
Select the schemas to be crawled.
The Crawl option is selected by default. To perform both operations, select the Crawl & Profile radio button.
Click Run to collect metadata from the connected source and load it into the Data Catalog.
After a successful crawl, the information appears in the Data Catalog > Databases tab.

The Schedule checkbox allows automated crawling and profiling at defined intervals, from a minute to a year.

Click the Schedule checkbox to enable the Select Period drop-down.
Select a time interval for the operation from the drop-down menu.
Click Schedule to initiate metadata collection from the connected source.
The system will automatically execute the selected operation (Crawl or Crawl & Profile) at the scheduled time.

Other Operations

The Connectors page provides a centralized view of all configured connectors, along with their health status.

Managing connectors includes:

Connectors Health: Displays the current status of each connector using a green icon for active connections and a red icon for inactive connections, helping to monitor the connectivity with data sources.
Viewing: Click the Eye icon next to the connector name to view connector details, including databases, tables, and columns.

Nine Dots Menu Options:

To view, edit, validate, build lineage, configure, or delete connectors, click on the Nine Dots menu.

Edit Connector: Update and revalidate the data source.
Validate Connector: Check the connection's integrity.
Settings: Modify connector settings.
- Crawler: Configure data extraction.
- Profiler: Customize data profiling rules and methods.
- Query Policies: Define query execution rules based on roles.
- Access Instructions: Add notes on how data can be accessed.
- Business Glossary Settings: Manage term associations at the connector level.
- Anomaly Detection Settings: Configure anomaly detection preferences at the connector level.
- Connection Pooling: Configure database connection pool parameters to optimize database connectivity and resource utilization. For more details, click here.
- Others: Configure notification recipients for metadata changes.
Build Lineage: Automatically build data lineage using source code parsing.
Delete Connector: Remove a connector with confirmation.

For more details on connector settings, click here.

Troubleshooting

If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.

S. No

Error Message

Error Description & Resolution

Error occurred while securely connecting to the CLIENT BRIDGE

The issue occurs when the Client Bridge connection is interrupted or the query execution exceeds the configured timeout. Differences in the Java version across Client Bridges may also affect job stability.

Resolution:

Verify Client Bridge connectivity and review the Bridge and job logs for connection interruptions or timeout-related errors.
Check the query execution time and data volume of the affected tables, and validate the applicable query/job timeout configuration.
Ensure the Client Bridge uses a supported and consistent Java version, and review the Bridge bootstrap configuration for any custom VM arguments.

Connection validation fails when the source requires encrypted communication

Validation fails when the source requires SSL but SSL enabled is set to false.

Resolution:

Set SSL enabled to true in the connection setup form.
Revalidate the connection.
If validation still fails, confirm whether additional network-level access is required.

Connection validation fails after adding a custom parameter or after a source-side configuration change

The source may require an additional connection parameter that is not available in the standard connection fields.

Resolution:

Review the validation error for the required parameter.
Add the required parameter manually to the connection string.
Revalidate the connection.

Connection validation fails when a credential manager other than the default database option is being used

Validation can fail when the configured key name does not exactly match the key in HashiCorp, AWS Secrets Manager, or Azure Key Vault.

Resolution:

Confirm that the dependent secret manager connector is created and correctly referenced under Credential Manager.
Verify the key name against the secret store.
Enter the correct key name and revalidate the connection

Query sheet execution takes longer than expected and eventually times out

The configured query timeout is too low for the volume of data being queried.

Resolution:

Increase Query Timeout Seconds under connection pooling settings, for example, from the default 180 seconds to a higher value.
If the same delay occurs during initial validation, increase the Connection Timeout value.

Password or other sensitive columns not visible when querying certain internal tables through the query sheet using the internal -1 connection

Certain sensitive tables and columns are intentionally restricted through the internal -1 MySQL connection because it has broad access to internal application data. Resolution:

Confirm whether the queried table is a restricted internal table, such as connection information.
Treat hidden sensitive columns, such as passwords, as expected restricted behavior rather than an error.

S. No

Error Message

Error Description & Resolution

Some databases or schemas missing after a crawl, with no error shown in the job logs

Include or exclude filters may unintentionally exclude a database or schema.

Resolution:

Run SHOW DATABASES on the source system to identify available databases.
Compare the results with the schema, table, and column filters under Crawler Rules.
Update the filters and re-run the crawl

Certain tables or functions expected in the catalog do not appear after a crawl

The required source permission may be missing for the affected object type.

Resolution:

Confirm USAGE permission on the information schema for schema-level crawling.
Confirm SELECT permission on the tables view for tables, columns, and views.
Confirm EXECUTE permission on the routines view for functions and stored procedures, and TRIGGER permission for triggers.

Primary key and foreign key relationships not appearing between tables

Relationship metadata cannot be retrieved when the required permissions are missing on the key column usage view.

Resolution:

Confirm SELECT and REFERENCES permissions on the key column usage view of the information schema.
Apply the required GRANT statement on the affected schema.
Re-run the crawl and verify that relationships appear.

FAQs

S. No

Question

Answer

What authentication types are supported for this connector?

Two authentication types are supported:

User ID and Password: Requires server, port, database, username, and password.
Azure Entra ID: Requires server, port, database, and username; no password is required.
For Entra ID, enter a normal username instead of an email address.
The username must already be mapped to the MySQL instance.

Are the source permissions different for the two authentication types?

No. Both authentication types require the same source permissions.

The authentication method only changes how the user is verified.
Required permissions for crawling and profiling remain the same.

What is the default port used for the MySQL connection?

The default MySQL port is 3306.

If the source uses a different port, enter the configured port in the connection setup form.

What is SSL enabled used for, and when should it be enabled?

SSL-enabled controls encrypted communication between the application and MySQL.

Set it to true when the source requires encrypted communication.
SSL may also be required when the source is accessed through a private network or private link.
Revalidate the connection after enabling SSL.

What is the SSH tunnel option, and when is it needed?

SSH tunneling is used when the MySQL server is behind a private network or firewall and cannot be accessed directly.

Enable SSH tunneling to display the SSH username, SSH port, host, and local port fields.
SSH establishes access to the private network first.
The MySQL server, port, username, and password are then used to connect to the database.

What is the internally created -1 MySQL connection used for?

The -1 connection is automatically created using the database credentials configured in the application properties file.

Supports the reporting framework, including dashboards and generated reports.
Is created or updated when the application starts or restarts.
Uses credentials from the properties file, including credentials retrieved from a configured secret manager.
Requires an application restart for credential changes to take effect.
Does not require manual creation or refresh through a reporting job.

Why are some tables or sensitive columns restricted when using the -1 connection?

The -1 connection has broad access to internal application data, so certain sensitive tables and columns are restricted.

Sensitive columns, such as passwords in connection information, are hidden from query results.
Other non-sensitive columns may remain visible.
This restriction applies specifically to the -1 connection.

S. No

Question

Answer

What permissions are required on the MySQL source system?

The following permissions are required:

USAGE on the information schema for schema-level crawling.
SELECT on the tables view for tables, columns, and views.
EXECUTE on the routines view for functions and stored procedures.
TRIGGER permission for trigger crawling.
SELECT and REFERENCES on the key column usage view for relationship crawling.

If a permission is missing, only the related object type is affected; other objects continue to be retrieved normally.

PreviousMicrosoft Access

NextMySQL - Lineage

Last updated 5 days ago

Was this helpful?