Apache Kafka
This article outlines the integration with the Apache Kafka connector, enabling streamlined metadata management through features such as crawling, data preview, and manual lineage building. It also ensures secure authentication via Credential Manager.
Overview
Connector Details
Connector Category
Messaging/Streaming
OvalEdge Release Supported
Release6.x to Release7.x
Connectivity
[How the connection is established with the Apache Kafka system]
Apache Java SDK
Connector Features
Crawling
✅
Delta Crawling
❌
Profiling
❌
Query Sheet
❌
Data Preview
✅
Auto Lineage
NA
Manual Lineage
✅
Secure Authentication via Credential Manager
✅
Data Quality
❌
DAM (Data Access Management)
✅
Bridge
✅
'NA' indicates that the respective feature is 'Not Applicable.'
Metadata Mapping
The following objects are crawled from Apache Kafka and mapped to the corresponding UI assets.
Schema
Cluster Name
Database Name
Database
Schema
Topics
Topic Name
Table
Tables
Table
Topics
Topic Data Type
Type
Tables
Table
Topics
Topic Comments
Source Description
Descriptions
Source Description
Messages
Message Key
Column
Table Columns
-
Messages
Message Type
Column Type
Table Columns
-
Set up a Connection
Prerequisites
The following are the prerequisites to establish a connection:
Service Account User Permissions
It is recommended to use a separate service account to establish the connection to the data source, configured with the following minimum set of permissions.
👨💻 Who can provide these permissions? These permissions are typically granted by the Apache Kafka administrator, as users may not have the required access to assign them independently.
Cluster
-
DESCRIBE
Topic
-
Topic: DESCRIBE, Topic: READ
Schema Registry
-
READ
Messages
-
READ on the Topic Resource
Connection Configuration Steps
Users are required to have the Connector Creator role in order to configure a new connection.
Log in to OvalEdge, go to Administration > Connectors, click + (New Connector), search for Apache Kafka, and complete the required parameters.
Fields marked with an asterisk (*) are mandatory for establishing a connection.
Connector Type
By default, "Kafka" is displayed as the selected connector type.
License Add Ons
Select the checkbox for Data Access Add-On to enable data access functionality.
Credential Manager*
Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.
Supported Credential Managers:
Database
HashiCorp
AWS Secrets Manager
Azure Key Vault
Cluster Authentication Type*
The following three types of authentication are supported for the Apache Kafka Server:
JAAS Config Path
App/Secret Key Credentials
Confluent Without Auth
Connector Description
Enter a brief description of the connector.
Connector Environment
Select the environment (Example: PROD, STG) configured for the connector.
Connector Name*
Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
Broker URL*
Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).
Cluster Name*
Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.
Cluster Id*
Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management.
Consumer Group Id
Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"
JAAS Config Path*
Enter the File path to the JAAS (Java Authentication and Authorization Service) configuration file. Supports PLAIN and Kerberos (KRB5LoginModule)
Kerberos: Sets java.security.auth.login.config system property
PLAIN: Reads file contents and sets SASL_JAAS_CONFIG
Security Protocol
Enter the security protocol for broker communication.
Ex:
PLAINTEXT: Unencrypted and unauthenticated communication.
SSL: Secure communication using SSL/TLS encryption.
SASL_PLAINTEXT: Authentication using SASL (Simple Authentication and Security Layer) without encryption.
SASL_SSL: Authentication using SASL with SSL/TLS encryption.
SASL Mechanism
Enter the SASL mechanism for authentication
Ex:
PLAIN: Plaintext username and password authentication.
SCRAM: Salted Challenge Response Authentication Mechanism, available as SCRAM-SHA-256 and SCRAM-SHA-512.
GSSAPI: Kerberos-based authentication.
OAUTHBEARER: OAuth 2.0 bearer token authentication.
Registry URL
Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081).
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
Schema Registry User
Enter the username for Schema Registry basic authentication.
Schema Registry Password
Enter the password for the Schema Registry basic authentication.
KRB5 Config Path
Enter the file path to the Kerberos configuration file (krb5.conf).
Connector Description
Enter a brief description of the connector.
Connector Environment
Select the environment (Example: PROD, STG) configured for the connector.
Connector Name*
Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
Broker URL*
Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).
Cluster Name*
Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.
Cluster Id*
Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management.
Consumer Group Id
Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"
App Key*
Enter the application key (username) for SASL PLAIN authentication.
Secret Key*
Enter the secret key (password) for SASL PLAIN authentication.
Security Protocol
Enter the security protocol for broker communication.
Ex:
PLAINTEXT: Unencrypted and unauthenticated communication.
SSL: Secure communication using SSL/TLS encryption.
SASL_PLAINTEXT: Authentication using SASL (Simple Authentication and Security Layer) without encryption.
SASL_SSL: Authentication using SASL with SSL/TLS encryption.
SASL Mechanism
Enter the SASL mechanism for authentication
Ex:
PLAIN: Plaintext username and password authentication.
SCRAM: Salted Challenge Response Authentication Mechanism, available as SCRAM-SHA-256 and SCRAM-SHA-512.
GSSAPI: Kerberos-based authentication.
OAUTHBEARER: OAuth 2.0 bearer token authentication.
Registry URL
Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081).
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
Schema Registry User
Enter the username for Schema Registry basic authentication.
Schema Registry Password
Enter the password for the Schema Registry basic authentication.
KRB5 Config Path
Enter the file path to the Kerberos configuration file (krb5.conf).
Connector Description
Enter a brief description of the connector.
Connector Environment
Select the environment (Example: PROD, STG) configured for the connector.
Connector Name*
Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
Broker URL*
Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).
Cluster Name*
Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.
Cluster Id*
Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management.
Consumer Group Id
Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"
Registry URL
Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081).
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
Default Governance Roles
Default Governance Roles*
Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection.
Admin Roles
Integration Admins*
Select one or more users from the dropdown list for Integration Admin and Security & Governance Admin. All users configured in the security settings are available for selection.
Bridge
Select Bridge*
If applicable, select the bridge from the drop-down list.
The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.
After entering all connection details, the following actions can be performed:
Click Validate to verify the connection.
Click Save to store the connection for future use.
Click Save & Configure to apply additional settings before saving.
The saved connection will appear on the Connectors home page.
Manage Connector Operations
Crawl/Profile
To perform crawl and profile operations, users must be assigned the Integration Admin role.
The Crawl/Profile button allows users to select one or more schemas for crawling and profiling.
Navigate to the Connectors page and click Crawl/Profile.
Select the schemas to be crawled.
The Crawl option is selected by default. To perform both operations, select the Crawl & Profile radio button.
Click Run to collect metadata from the connected source and load it into the Data Catalog.
After a successful crawl, the information appears in the Data Catalog > Databases tab.
The Schedule checkbox allows automated crawling and profiling at defined intervals, from a minute to a year.
Click the Schedule checkbox to enable the Select Period drop-down.
Select a time interval for the operation from the drop-down menu.
Click Schedule to initiate metadata collection from the connected source.
The system will automatically execute the selected operation (Crawl or Crawl & Profile) at the scheduled time.
Other Operations
The Connectors page provides a centralized view of all configured connectors, along with their health status.
Managing connectors includes:
Connectors Health: Displays the current status of each connector with a green icon for active connections and a red icon for inactive connections, helping monitor connectivity to data sources.
Viewing: Click the Eye icon next to the connector name to view connector details, including databases, tables, columns, and codes.
Nine Dots Menu Options:
To view, edit, validate, configure, or delete connectors, click on the Nine Dots menu.
Edit Connector: Update and revalidate the data source.
Validate Connector: Check the connection's integrity.
Settings: Modify connector settings.
Crawler: Configure data extraction.
Profiler: Customize data profiling rules and methods.
Access Instructions: Add notes on how data can be accessed.
Business Glossary Settings: Manage term associations at the connector level.
Others: Configure notification recipients for metadata changes.
Delete Connector: Remove a connector with confirmation.
Connectivity Troubleshooting
If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.
1
TimeoutException / REQUEST_TIMED_OUT (code 7)
Error Description:
The request did not complete within the configured time.
Resolution:
Increase timeout settings, such as request.timeout.ms, max.block.ms, and session.timeout.ms. Check broker responsiveness, network latency, and system load.
2
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed due to: Invalid credentials
Error Description:
The client attempted SASL-based login (SCRAM or PLAIN), but the broker rejected the credentials.
Resolution:
Verify the username and password. Confirm SCRAM credentials on the broker using kafka-configs.sh. Ensure the client’s sasl.mechanism matches the broker configuration and that the user exists and is configured correctly.
3
SSL certificate handshake / TLS Auth error (when using SSL or SASL_SSL)
Error Description:
The client could not establish an SSL/TLS connection with the broker due to certificate or hostname issues.
Resolution:
Provide the correct truststore that includes the CA certificate used by the broker. Check the ssl.endpoint.identification.algorithm setting. Ensure the broker certificate’s SAN matches the client's hostname.
4
HTTP Status 403 Forbidden
Error Description:
The client connected to the Schema Registry but does not have permission to perform the requested action.
Resolution:
Verify that the user or principal has the required ACLs for the Schema Registry resource. Review Schema Registry logs for denied operations. Confirm that basic.auth.user.info is correct and that the account has the necessary permissions.
5
SSL/TLS handshake failure when accessing Schema Registry over HTTPS
Error Description:
The client failed to establish an SSL/TLS connection to the Schema Registry.
Resolution:
Use the correct truststore or keystore. Validate the ssl.endpoint.identification.algorithm setting. Ensure that the Schema Registry URL hostname matches the certificate’s subject or SAN entries.
Copyright © 2026, OvalEdge LLC, Peachtree Corners, GA, USA.
Was this helpful?