# Apache Kafka

This article outlines the integration with the **Apache Kafka** connector, enabling streamlined metadata management through features such as crawling, profiling, data preview, and manual lineage building. This connector uses the **Apache Java SDK** to establish connectivity with **Apache Kafka clusters**. It supports **JAAS Config Path**, **App/Secret Key Credentials**, and **Confluent Without Auth** authentication methods for accessing Kafka topics and metadata objects.

<figure><img src="/files/DsjA0iK47eYNFn0LZduP" alt=""><figcaption></figcaption></figure>

## Overview

### Connector Details

| Connector Category                                                                          | Messaging/Streaming  |
| ------------------------------------------------------------------------------------------- | -------------------- |
| OvalEdge Release Supported                                                                  | Release6.x and later |
| <p>Connectivity</p><p>\[How the connection is established with the Apache Kafka system]</p> | Apache Java SDK      |

### Connector Features

| Feature                                      | Availability |
| -------------------------------------------- | :----------: |
| Crawling                                     |       ✅      |
| Delta Crawling                               |       ❌      |
| Profiling                                    |       ❌      |
| Query Sheet                                  |       ❌      |
| Data Preview                                 |       ✅      |
| Auto Lineage                                 |      NA      |
| Manual Lineage                               |       ✅      |
| Secure Authentication via Credential Manager |       ✅      |
| Data Quality                                 |       ❌      |
| DAM (Data Access Management)                 |       ✅      |
| Bridge                                       |       ✅      |

{% hint style="info" %}
'NA' indicates that the respective feature is 'Not Applicable.'
{% endhint %}

### Metadata Mapping

The following objects are crawled from Apache Kafka and mapped to the corresponding UI assets.

<table><thead><tr><th width="191.5">Apache Kafka Object</th><th width="201.5">Apache Kafka Attribute</th><th width="178">OvalEdge Attribute</th><th width="176.75">OvalEdge Category</th><th width="160.5">OvalEdge  Type</th></tr></thead><tbody><tr><td>Schema</td><td>Cluster Name</td><td>Database Name</td><td>Database</td><td>Schema</td></tr><tr><td>Topics</td><td>Topic Name</td><td>Table</td><td>Tables</td><td>Table</td></tr><tr><td>Topics</td><td>Topic Data Type</td><td>Type</td><td>Tables</td><td>Table</td></tr><tr><td>Topics</td><td>Topic Comments</td><td>Source Description</td><td>Descriptions</td><td>Source Description</td></tr><tr><td>Messages</td><td>Message Key</td><td>Column</td><td>Table Columns</td><td>-</td></tr><tr><td>Messages</td><td>Message Type</td><td>Column Type</td><td>Table Columns</td><td>-</td></tr></tbody></table>

## Set up a Connection

### Prerequisites

The following are the prerequisites to establish a connection:

### Service Account User Permissions

{% hint style="info" %}
It is recommended to use a separate service account to establish the connection to the data source, configured with the following minimum set of permissions.
{% endhint %}

{% hint style="info" %}
**👨‍💻 Who can provide these permissions?** These permissions are typically granted by the Apache Kafka administrator, as users may not have the required access to assign them independently.
{% endhint %}

| Objects         | Access Permission            |
| --------------- | ---------------------------- |
| Cluster         | DESCRIBE                     |
| Topic           | Topic: DESCRIBE, Topic: READ |
| Schema Registry | READ                         |
| Messages        | READ on the Topic Resource   |

## Connection Configuration Steps

{% hint style="info" %}
Users are required to have the Connector Creator role in order to configure a new connection.
{% endhint %}

1. Log in to OvalEdge, go to **Administration** > **Connectors**, click + **(New Connector)**, search for **Apache Kafka**, and complete the required parameters.

{% hint style="info" %}
Fields marked with an asterisk (\*) are mandatory for establishing a connection.
{% endhint %}

<table><thead><tr><th width="219.45452880859375">Field Name</th><th>Description</th></tr></thead><tbody><tr><td>Connector Type</td><td>By default, "<strong>Kafka</strong>" is displayed as the selected connector type.</td></tr><tr><td>License Add Ons</td><td>Select the checkbox for Data Access Add-On to enable data access functionality.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/license-types-and-add-ons">here</a>.</td></tr><tr><td>Credential Manager*</td><td><p>Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.</p><p>Supported Credential Managers:</p><ul><li>Database</li><li>HashiCorp</li><li>AWS Secrets Manager</li><li>Azure Key Vault</li></ul><p>For more details, click <a href="https://docs.ovaledge.com/connectors/additional-requirements/credential-manager-configuration">here</a>.</p></td></tr><tr><td>Cluster Authentication Type*</td><td><p>The following three types of authentication are supported for Apache Kafka:</p><ul><li>JAAS Config Path</li><li>App/Secret Key Credentials</li><li>Confluent Without Auth</li></ul></td></tr></tbody></table>

{% tabs %}
{% tab title="JAAS Config Path" %}

<table><thead><tr><th width="196.272705078125">Field Name</th><th>Description</th></tr></thead><tbody><tr><td>Connector Description</td><td>Enter a brief description of the connector.</td></tr><tr><td>Connector Environment</td><td>Select the environment (Example: PROD, STG) configured for the connector.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/prerequisites#connector-environment">here</a>.</td></tr><tr><td>Connector Name*</td><td><p>Enter a unique name for the Apache Kafka connection              </p><p>(Example: "ApacheKafkadb").</p></td></tr><tr><td>Broker URL*</td><td>Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).</td></tr><tr><td>Cluster Name*</td><td>Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.</td></tr><tr><td>Consumer Group Id</td><td>Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"</td></tr><tr><td>JAAS Config Path*</td><td><p>Enter the File path to the JAAS (Java Authentication and Authorization Service) configuration file. Supports PLAIN and Kerberos (KRB5LoginModule)</p><ul><li>Kerberos: Sets java.security.auth.login.config system property</li><li>PLAIN: Reads file contents and sets SASL_JAAS_CONFIG</li></ul></td></tr><tr><td>Security Protocol </td><td><p>Enter the security protocol for broker communication.</p><p>Ex: </p><ul><li>PLAINTEXT: Unencrypted and unauthenticated communication.</li><li>SSL: Secure communication using SSL/TLS encryption.</li><li>SASL_PLAINTEXT: Authentication using SASL (Simple Authentication and Security Layer) without encryption.</li><li>SASL_SSL: Authentication using SASL with SSL/TLS encryption.</li></ul></td></tr><tr><td>SASL Mechanism</td><td><p>Enter the SASL mechanism for authentication</p><p>Ex: </p><ul><li>PLAIN: Plaintext username and password authentication.</li><li>SCRAM: Salted Challenge Response Authentication Mechanism, available as SCRAM-SHA-256 and SCRAM-SHA-512.</li><li>GSSAPI: Kerberos-based authentication.</li><li>OAUTHBEARER: OAuth 2.0 bearer token authentication.</li></ul></td></tr><tr><td>Registry URL</td><td><p>Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081). </p><p>Used for Avro, Protobuf, and JSON Schema serialization/deserialization</p></td></tr><tr><td>Schema Registry User</td><td>Enter the username for Schema Registry basic authentication.</td></tr><tr><td>Schema Registry Password</td><td>Enter the password for the Schema Registry basic authentication.</td></tr><tr><td>KRB5 Config Path</td><td>Enter the file path to the Kerberos configuration file (krb5.conf). </td></tr></tbody></table>
{% endtab %}

{% tab title="App/Secret Key Credentials" %}

<table><thead><tr><th width="211.72723388671875">Field Name</th><th>Description</th></tr></thead><tbody><tr><td>Connector Description</td><td>Enter a brief description of the connector.</td></tr><tr><td>Connector Environment</td><td>Select the environment (Example: PROD, STG) configured for the connector.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/prerequisites#connector-environment">here</a>.</td></tr><tr><td>Connector Name*</td><td><p>Enter a unique name for the Apache Kafka connection              </p><p>(Example: "ApacheKafkadb").</p></td></tr><tr><td>Broker URL*</td><td>Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).</td></tr><tr><td>Cluster Name*</td><td>Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.</td></tr><tr><td>Cluster Id*</td><td>Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management.</td></tr><tr><td>Consumer Group Id</td><td>Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"</td></tr><tr><td>App Key*</td><td>Enter the application key (username) for SASL PLAIN authentication. </td></tr><tr><td>Secret Key*</td><td>Enter the secret key (password) for SASL PLAIN authentication. </td></tr><tr><td>Security Protocol </td><td><p>Enter the security protocol for broker communication.</p><p>Ex: </p><ul><li>PLAINTEXT: Unencrypted and unauthenticated communication.</li><li>SSL: Secure communication using SSL/TLS encryption.</li><li>SASL_PLAINTEXT: Authentication using SASL (Simple Authentication and Security Layer) without encryption.</li><li>SASL_SSL: Authentication using SASL with SSL/TLS encryption.</li></ul></td></tr><tr><td>SASL Mechanism</td><td><p>Enter the SASL mechanism for authentication</p><p>Ex: </p><ul><li>PLAIN: Plaintext username and password authentication.</li><li>SCRAM: Salted Challenge Response Authentication Mechanism, available as SCRAM-SHA-256 and SCRAM-SHA-512.</li><li>GSSAPI: Kerberos-based authentication.</li><li>OAUTHBEARER: OAuth 2.0 bearer token authentication.</li></ul></td></tr><tr><td>Registry URL</td><td><p>Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081). </p><p>Used for Avro, Protobuf, and JSON Schema serialization/deserialization</p></td></tr><tr><td>Schema Registry User</td><td>Enter the username for Schema Registry basic authentication.</td></tr><tr><td>Schema Registry Password</td><td>Enter the password for the Schema Registry basic authentication.</td></tr><tr><td>KRB5 Config Path</td><td>Enter the file path to the Kerberos configuration file (krb5.conf). </td></tr></tbody></table>
{% endtab %}

{% tab title="Confluent Without Auth" %}

<table><thead><tr><th width="212.6363525390625">Field Name</th><th>Description</th></tr></thead><tbody><tr><td>Connector Description</td><td>Enter a brief description of the connector.</td></tr><tr><td>Connector Environment</td><td>Select the environment (Example: PROD, STG) configured for the connector.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/prerequisites#connector-environment">here</a>.</td></tr><tr><td>Connector Name*</td><td><p>Enter a unique name for the Apache Kafka connection              </p><p>(Example: "ApacheKafkadb").</p></td></tr><tr><td>Broker URL*</td><td>Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092).</td></tr><tr><td>Cluster Name*</td><td>Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge.</td></tr><tr><td>Consumer Group Id</td><td>Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11"</td></tr><tr><td>Registry URL</td><td><p>Enter the URL of the Schema Registry (e.g., https://schema-registry.example.com:8081). </p><p>Used for Avro, Protobuf, and JSON Schema serialization/deserialization</p></td></tr></tbody></table>
{% endtab %}
{% endtabs %}

**Default Governance Roles**

<table data-header-hidden><thead><tr><th width="218.54541015625"></th><th></th></tr></thead><tbody><tr><td>Default Governance Roles*</td><td>Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection.</td></tr></tbody></table>

**Admin Roles**

<table data-header-hidden><thead><tr><th width="218.54547119140625"></th><th></th></tr></thead><tbody><tr><td>Integration Admins*</td><td>Select one or more users from the dropdown list for Integration Admin and Security &#x26; Governance Admin. All users configured in the security settings are available for selection.</td></tr></tbody></table>

**Bridge**

<table data-header-hidden><thead><tr><th width="218.54547119140625"></th><th></th></tr></thead><tbody><tr><td>Select Bridge*</td><td><p>If applicable, select the bridge from the drop-down list.</p><p>The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.</p></td></tr></tbody></table>

2. After entering all connection details, the following actions can be performed:
   1. Click **Validate** to verify the connection.
   2. Click **Save** to store the connection for future use.
   3. Click **Save & Configure** to apply additional settings before saving.
3. The saved connection will appear on the **Connectors home** page.

## Manage Connector Operations

### Crawl/Profile

{% hint style="info" %}
To perform crawl and profile operations, users must be assigned the Integration Admin role.
{% endhint %}

The **Crawl/Profile** button allows users to select one or more schemas for crawling and profiling.

1. Navigate to the Connectors page and click **Crawl/Profile**.
2. Select the schemas to be crawled.
3. The **Crawl** option is selected by default. To perform both operations, select the **Crawl & Profile** radio button.
4. Click **Run** to collect metadata from the connected source and load it into the **Data Catalog**.
5. After a successful crawl, the information appears in the **Data Catalog > Databases** tab.

The **Schedule** checkbox allows automated crawling and profiling at defined intervals, from a minute to a year.

1. Click the **Schedule** checkbox to enable the **Select Period** drop-down.
2. Select a time interval for the operation from the drop-down menu.
3. Click **Schedule** to initiate metadata collection from the connected source.
4. The system will automatically execute the selected operation (**Crawl** or **Crawl & Profile**) at the scheduled time.

### Other Operations

The **Connectors** page provides a centralized view of all configured connectors, along with their health status.

**Managing connectors includes**:

* **Connectors Health**: Displays the current status of each connector with a **green** icon for active connections and a **red** icon for inactive connections, helping monitor connectivity to data sources.
* **Viewing**: Click the **Eye** icon next to the connector name to view connector details, including databases, tables, columns, and codes.

**Nine Dots Menu Options**:

To view, edit, validate, configure, or delete connectors, click on the Nine Dots menu.

* **Edit Connector**: Update and revalidate the data source.
* **Validate Connector**: Check the connection's integrity.
* **Settings**: Modify connector settings.
  * **Crawler**: Configure data extraction.
  * **Profiler**: Customize data profiling rules and methods.
  * **Access Instructions**: Add notes on how data can be accessed.
  * **Business Glossary Settings**: Manage term associations at the connector level.
  * **Others**: Configure notification recipients for metadata changes.
* **Delete Connector**: Remove a connector with confirmation.

For more details, click [here](https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/connector-settings).

### Connectivity Troubleshooting

If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.

<table><thead><tr><th width="82.3333740234375">S.No.</th><th width="275.3333740234375">Error Message(s)</th><th>Error Description &#x26; Resolution</th></tr></thead><tbody><tr><td>1</td><td>TimeoutException / REQUEST_TIMED_OUT (code 7)</td><td><p>Error Description: </p><p>The request did not complete within the configured time.</p><p>Resolution:</p><p>Increase timeout settings, such as request.timeout.ms, max.block.ms, and session.timeout.ms. Check broker responsiveness, network latency, and system load.</p></td></tr><tr><td>2</td><td>org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed due to: Invalid credentials</td><td><p>Error Description: </p><p>The client attempted SASL-based login (SCRAM or PLAIN), but the broker rejected the credentials.</p><p>Resolution:</p><p>Verify the username and password. Confirm SCRAM credentials on the broker using kafka-configs.sh. Ensure the client’s sasl.mechanism matches the broker configuration and that the user exists and is configured correctly.</p></td></tr><tr><td>3</td><td>SSL certificate handshake / TLS Auth error (when using SSL or SASL_SSL)</td><td><p>Error Description: </p><p>The client could not establish an SSL/TLS connection with the broker due to certificate or hostname issues.</p><p>Resolution:</p><p>Provide the correct truststore that includes the CA certificate used by the broker. Check the ssl.endpoint.identification.algorithm setting. Ensure the broker certificate’s SAN matches the client's hostname.</p></td></tr><tr><td>4</td><td>HTTP Status 403 Forbidden</td><td><p>Error Description: </p><p>The client connected to the Schema Registry but does not have permission to perform the requested action.</p><p>Resolution:</p><p>Verify that the user or principal has the required ACLs for the Schema Registry resource. Review Schema Registry logs for denied operations. Confirm that basic.auth.user.info is correct and that the account has the necessary permissions.</p></td></tr><tr><td>5</td><td>SSL/TLS handshake failure when accessing Schema Registry over HTTPS</td><td><p>Error Description: </p><p>The client failed to establish an SSL/TLS connection to the Schema Registry.</p><p>Resolution:</p><p>Use the correct truststore or keystore. Validate the ssl.endpoint.identification.algorithm setting. Ensure that the Schema Registry URL hostname matches the certificate’s subject or SAN entries.</p></td></tr></tbody></table>

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners, GA, USA.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.ovaledge.com/release8.1/connectors/connector-repositories/messaging-streaming/apache-kafka.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.