Import data from Kafka

Create a Kafka connector, select topics in an import task, and write messages to structured Catalog tables. The connector owns connection and authentication settings; consumer groups, topics, and starting positions belong to the import task.

Prepare the connection

Prepare a reachable cluster address, credentials, topic access, and a writable target database. Prepare Schema Registry settings if required.

  1. Open Data Connections > Connectors and create a Kafka connector.

  2. Enter bootstrap_servers and select a security protocol.

  3. Provide the matching TLS or SASL settings.

  4. Configure Schema Registry when used.

  5. Test the connection and save.

Security protocol

Checks needed

PLAINTEXT

Network/cluster access scope for unencrypted connections.

SSL

TLS connection and the required CA certificate.

SASL_PLAINTEXT

SASL mechanism, username, and password.

SASL_SSL

SASL authentication and TLS certificate settings.

SASL mechanisms include PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. Enter credentials in their designated fields, not in the connector name, description, or screenshots.

Select a topic and create an import task

  1. Create a structured import task using the connector.

  2. Discover and select the topic; inspect samples or available field information.

  3. Configure the consumer group and starting position in the task form.

  4. Select or create a destination table and check field and type mappings.

  5. Submit, inspect task status, and query a recognizable test message in the destination.

  6. Check the run state in task details, then query the Catalog target table for a recognizable test message.

A connection test does not validate topic reads or table writes. A sample does not prove that every message has the same schema. Use the form, capability results, and errors to determine supported formats.

Check and recover

For connection failures, check the address, security protocol, certificates, SASL mechanism, and network access. For invisible topics, check topic names and access permissions. For target-table write failures, check field mappings, data types, and write permissions.

Before changing the consumption starting position or recreating a task, inspect existing runs and target data to avoid duplicate writes from repeated consumption. Use pause, resume, and retry actions actually available in task details. Recreating tasks is not the default response to every failure.

For programmatic operations, see the SDK guide Check Data Sources and Read Samples.

Last updated on