Kafka integration

Overview

Verta supports near real-time/asynchronous models with Kafka integration, allowing system admins to seamlessly connect their Kafka streaming servers to Verta. Once integrated, Verta endpoints can be configured to read data from a Kafka topic, perform predictions, and write the output to another Kafka topic. In case of any error scenarios, Verta can send the errors to an error topic for further analysis or handling.

Setup Kafka integration

Step 1

If you are a system admin, go to the Integrations tab in the system admin panel to set up Kafka integration.

Step 2

During the setup process, you will be prompted to provide the following information for the Kafka integration:

  • Integration Name: Choose a descriptive name for future reference. Broker Address: Enter the address of the Kafka broker, typically in the format "BROKER1:PORT1". The broker allows consumers to fetch messages based on topic, partition, and offset.

  • TLS (Transport Layer Security): Optionally enable TLS for server authentication and encryption.

  • Kerberos: Optionally enable Kerberos network authentication protocol to establish a connection between Verta and the Kafka cluster.

These details will be requested when setting up the Kafka integration in Verta and given below is sample integration.

Setup TLS

When TLS (Transport Layer Security) is enabled, you will need to upload the certificate for the Verta system to establish a secure connection. You have the option to either upload the certificate file or paste the certificate content directly into the provided field.

Setup Kerberos

When Kerberos is enabled, you will need to provide the following information:

  • Client Name: Enter the client name associated with the Kerberos authentication.

  • Service Name: Specify the service name to be used for the connection.

  • krb5.conf File: This file contains the configuration information for Kerberos. You can either upload the krb5.conf file or paste its contents into the provided field.

  • Keytab: The Keytab file stores your keys and is typically a binary file. You can either upload the Keytab file or paste its contents.

Step 3

Test the connection before saving.

Note: To ensure a successful connection between Verta and the Kafka cluster, you can use the "Run Test" button. The integration cannot be saved unless the test is successful, indicating a valid connection.

After successfully testing and saving the setup, the Kafka integration will become active. This means that all users within your organization will have the ability to create endpoints with the Kafka setup. They can utilize the integration to establish connections with Kafka topics, perform predictions, and write output.

To learn more about deploying a model with Kafka, you can click on this link.

Note: Every endpoint should be uniquely associated with an input, output and error topic. Topics should not be re-used across endpoints.

Kafka error troubleshooting

Here is a troubleshooting guide for possible error scenarios.

Kafka cluster down or unavailable

Error: join and sync loop errored Resolution: Kafka cluster is down and contact your Kafka system administrators.

Broker(s) unavailable

Error: unable to open connection to broker Resolution: Confirm broker address and port are correct and if they look good contact your Kafka system administrators.

Kafka topic doesn’t exist

Error: kafka topic %s not found Resolution: Create the Kafka topic rr tell the data scientist that this topic no longer exists and they should use a different topic.

Offset out of range

Error: offset out of range Resolution: Confirm the topic consumption settings are correct and contact your Kafka system administrators.

TLS permission errors

Error: tls: first record does not look like a TLS handshake Resolution: Confirm the port is correct for the broker. Noet that TLS and plaintext use different ports. If not resolved, contact Kafka system administrators.

Kerberos errors

Error: "Kafka configuration error: Kafka validation failed (check client name in kerberos conf?): : could not get valid TGT for client's realm: [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database - CLIENT_NOT_FOUND" when testing kafka configuration Resolution: Confirm the kerberos client name is correct. If not resolved, contact Kafka system administrators.

Error: "Kafka configuration error: Kafka validation failed (check realms and server name in kerberos conf?): : [Root cause: KDC_Error] KDC_Error: TGS Exchange Error: kerberos error response from KDC when requesting for kafk/integrations--kafka-kafka-0.integrations--kafka-kafka-headless.ci.svc.cluster.local: KRB Error: (7) KDC_ERR_S_PRINCIPAL_UNKNOWN Server not found in Kerberos database - LOOKING_UP_SERVER" when testing kafka configuration Resolution: Confirm the kerberos service name is correct. If not resolved, contact Kafka system administrators.

Error: "Kafka configuration error: Kafka validation failed (check client name in kerberos conf?): : could not get valid TGT for client's realm: [Root cause: Networking_Error] Networking_Error: AS Exchange Error: failed sending AS_REQ to KDC: failed to communicate with KDC. Attempts made with UDP (error sending to a KDC: error resolving KDC address: lookup integrations--kafka-kdc-server-0.integrations--kafka-kdc-server-headless.ci.svc.cluster.locl on 172.20.0.10:53: no such host) and then TCP (error sending to a KDC: error resolving KDC address: lookup integrations--kafka-kdc-server-0.integrations--kafka-kdc-server-headless.ci.svc.cluster.locl on 172.20.0.10:53: no such host)" when testing kafka configuration Resolution: Confirm the kerberos krb5.conf file contents are correct for the kdc server. If not resolved, contact Kafka system administrators.

Error: "Kafka configuration error: got code 400 and message \"failed to base64-decode keytab: \"illegal base64 data at input byte 104\"\" when testing kafka configuration Resolution: Confirm the kerberos keytab contents are valid. If not resolved, contact Kafka system administrators.

Serialization/deserialization errors

Resolution: Reformat input data so it matches the model’s requirements. Reminder that Verta models require json-formatted data.

Timeouts

Resolution:

  • The system is overloaded and needs to get through its backlog of predictions.

  • Provision additional resources.

  • Engage autoscaling in the kafka integration to allow multiple kafka consumer pods to service high volume topics

Last updated