Connectivity
\[How the connection is established with the Apache Kafka system]
| Apache Java SDK | ### Connector Features | Feature | Availability | | -------------------------------------------- | :----------: | | Crawling | ✅ | | Delta Crawling | ❌ | | Profiling | ❌ | | Query Sheet | ❌ | | Data Preview | ✅ | | Auto Lineage | NA | | Manual Lineage | ✅ | | Secure Authentication via Credential Manager | ✅ | | Data Quality | ❌ | | DAM (Data Access Management) | ✅ | | Bridge | ✅ | {% hint style="info" %} 'NA' indicates that the respective feature is 'Not Applicable.' {% endhint %} ### Metadata Mapping The following objects are crawled from Apache Kafka and mapped to the corresponding UI assets.| Apache Kafka Object | Apache Kafka Attribute | OvalEdge Attribute | OvalEdge Category | OvalEdge Type |
|---|---|---|---|---|
| Schema | Cluster Name | Database Name | Database | Schema |
| Topics | Topic Name | Table | Tables | Table |
| Topics | Topic Data Type | Type | Tables | Table |
| Topics | Topic Comments | Source Description | Descriptions | Source Description |
| Messages | Message Key | Column | Table Columns | - |
| Messages | Message Type | Column Type | Table Columns | - |
-
-
-
-
Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.
Supported Credential Managers:
The following three types of authentication are supported for the Apache Kafka Server:
Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
| | Broker URL\* | Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092). | | Cluster Name\* | Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge. | | Cluster Id\* | Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management. | | Consumer Group Id | Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11" | | JAAS Config Path\* |Enter the File path to the JAAS (Java Authentication and Authorization Service) configuration file. Supports PLAIN and Kerberos (KRB5LoginModule)
Enter the security protocol for broker communication.
Ex:
Enter the SASL mechanism for authentication
Ex:
Enter the URL of the Schema Registry (e.g.,
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
| | Schema Registry User | Enter the username for Schema Registry basic authentication. | | Schema Registry Password | Enter the password for the Schema Registry basic authentication. | | KRB5 Config Path | Enter the file path to the Kerberos configuration file (krb5.conf). | | {% endtab %} | | {% tab title="App/Secret Key Credentials" %} | Field Name | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Connector Description | Enter a brief description of the connector. | | Connector Environment | Select the environment (Example: PROD, STG) configured for the connector. | | Connector Name\* |Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
| | Broker URL\* | Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092). | | Cluster Name\* | Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge. | | Cluster Id\* | Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management. | | Consumer Group Id | Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11" | | App Key\* | Enter the application key (username) for SASL PLAIN authentication. | | Secret Key\* | Enter the secret key (password) for SASL PLAIN authentication. | | Security Protocol |Enter the security protocol for broker communication.
Ex:
Enter the SASL mechanism for authentication
Ex:
Enter the URL of the Schema Registry (e.g.,
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
| | Schema Registry User | Enter the username for Schema Registry basic authentication. | | Schema Registry Password | Enter the password for the Schema Registry basic authentication. | | KRB5 Config Path | Enter the file path to the Kerberos configuration file (krb5.conf). | | {% endtab %} | | {% tab title="Confluent Without Auth" %} | Field Name | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Connector Description | Enter a brief description of the connector. | | Connector Environment | Select the environment (Example: PROD, STG) configured for the connector. | | Connector Name\* |Enter a unique name for the Apache Kafka connection
(Example: "ApacheKafkadb").
| | Broker URL\* | Enter the comma-separated list of Kafka broker host: port addresses (e.g., broker1:9092,broker2:9092). | | Cluster Name\* | Enter the logical name for the Kafka cluster (e.g., "Cluster1"). Used for organization and identification in OvalEdge. | | Cluster Id\* | Enter the unique identifier for the Kafka cluster, often from Confluent Cloud or the Kafka provider. It is used for cluster identification and management. | | Consumer Group Id | Enter the consumer group identifier to coordinate consumers and manage offsets. If not provided, defaults to "OE-11" | | Registry URL |Enter the URL of the Schema Registry (e.g.,
Used for Avro, Protobuf, and JSON Schema serialization/deserialization
| | {% endtab %} | | | {% endtabs %} | | **Default Governance Roles** | Default Governance Roles\* | Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection. | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | **Admin Roles** | Integration Admins\* | Select one or more users from the dropdown list for Integration Admin and Security & Governance Admin. All users configured in the security settings are available for selection. | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | **Bridge** | Select Bridge\* |If applicable, select the bridge from the drop-down list.
The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.
| | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 2. After entering all connection details, the following actions can be performed: 1. Click **Validate** to verify the connection. 2. Click **Save** to store the connection for future use. 3. Click **Save & Configure** to apply additional settings before saving. 3. The saved connection will appear on the Connectors home page. ## Manage Connector Operations ### Crawl/Profile {% hint style="info" %} To perform crawl and profile operations, users must be assigned the Integration Admin role. {% endhint %} The **Crawl/Profile** button allows users to select one or more schemas for crawling and profiling. 1. Navigate to the Connectors page and click **Crawl/Profile**. 2. Select the schemas to be crawled. 3. The **Crawl** option is selected by default. To perform both operations, select the **Crawl & Profile** radio button. 4. Click **Run** to collect metadata from the connected source and load it into the **Data Catalog**. 5. After a successful crawl, the information appears in the **Data Catalog > Databases** tab. The **Schedule** checkbox allows automated crawling and profiling at defined intervals, from a minute to a year. 1. Click the **Schedule** checkbox to enable the **Select Period** drop-down. 2. Select a time interval for the operation from the drop-down menu. 3. Click **Schedule** to initiate metadata collection from the connected source. 4. The system will automatically execute the selected operation (**Crawl** or **Crawl & Profile**) at the scheduled time. ### Other Operations The **Connectors** page provides a centralized view of all configured connectors, along with their health status. **Managing connectors includes**: * **Connectors Health**: Displays the current status of each connector with a green icon for active connections and a red icon for inactive connections, helping monitor connectivity to data sources. * **Viewing**: Click the **Eye** icon next to the connector name to view connector details, including databases, tables, columns, and codes. **Nine Dots Menu Options**: To view, edit, validate, configure, or delete connectors, click on the Nine Dots menu. * **Edit Connector**: Update and revalidate the data source. * **Validate Connector**: Check the connection's integrity. * **Settings**: Modify connector settings. * **Crawler**: Configure data extraction. * **Profiler**: Customize data profiling rules and methods. * **Access Instructions**: Add notes on how data can be accessed. * **Business Glossary Settings**: Manage term associations at the connector level. * **Others**: Configure notification recipients for metadata changes. * **Delete Connector**: Remove a connector with confirmation. ### Connectivity Troubleshooting If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.| S.No. | Error Message(s) | Error Description & Resolution |
|---|---|---|
| 1 | TimeoutException / REQUEST_TIMED_OUT (code 7) | Error Description: The request did not complete within the configured time. Resolution: Increase timeout settings, such as request.timeout.ms, max.block.ms, and session.timeout.ms. Check broker responsiveness, network latency, and system load. |
| 2 | org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed due to: Invalid credentials | Error Description: The client attempted SASL-based login (SCRAM or PLAIN), but the broker rejected the credentials. Resolution: Verify the username and password. Confirm SCRAM credentials on the broker using kafka-configs.sh. Ensure the client’s sasl.mechanism matches the broker configuration and that the user exists and is configured correctly. |
| 3 | SSL certificate handshake / TLS Auth error (when using SSL or SASL_SSL) | Error Description: The client could not establish an SSL/TLS connection with the broker due to certificate or hostname issues. Resolution: Provide the correct truststore that includes the CA certificate used by the broker. Check the ssl.endpoint.identification.algorithm setting. Ensure the broker certificate’s SAN matches the client's hostname. |
| 4 | HTTP Status 403 Forbidden | Error Description: The client connected to the Schema Registry but does not have permission to perform the requested action. Resolution: Verify that the user or principal has the required ACLs for the Schema Registry resource. Review Schema Registry logs for denied operations. Confirm that basic.auth.user.info is correct and that the account has the necessary permissions. |
| 5 | SSL/TLS handshake failure when accessing Schema Registry over HTTPS | Error Description: The client failed to establish an SSL/TLS connection to the Schema Registry. Resolution: Use the correct truststore or keystore. Validate the ssl.endpoint.identification.algorithm setting. Ensure that the Schema Registry URL hostname matches the certificate’s subject or SAN entries. |