Kafka Direct

Stream messages that you write into topics in your Streamkap Kafka through Streamkap pipelines to your destinations.

Kafka Direct turns topics in your Streamkap Kafka into a source. You produce messages into those topics, and Streamkap streams them through pipelines to your destinations. Unlike CDC-based sources that capture changes from a database transaction log, Kafka Direct simply forwards the messages on the topics you list.

Prerequisites

• On a dedicated Kafka cluster: a Kafka user created in Kafka Access with WRITE and DESCRIBE permissions on your topics, used by your producer. See Kafka (Writing) for the permissions and connection settings. Kafka Access doesn’t create new users on the shared cluster.
• The names of the topics you write to

Key Differences from CDC Sources

Kafka Direct behaves differently from CDC-based sources (such as PostgreSQL, MySQL, or MongoDB):

• No transaction log – Kafka Direct reads messages directly from Kafka topics rather than tailing a database transaction log.
• No snapshots – There is no initial snapshot or backfill mechanism.
• No heartbeats – Heartbeat monitoring is not applicable to Kafka Direct sources.
• Topic naming – Topics keep exactly the names you list; Streamkap adds no prefix (CDC sources use source_{id}).

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Source

• Navigate to Add Connectors.
• Choose Kafka Direct.

2. Settings

• Name: Enter a unique and memorable name for your connector.

• Topic prefix: The prefix your topic names start with, such as myprefix. New topics in your Streamkap Kafka whose names start with it are added to the source automatically. Streamkap does not add it to topic names.

• Topics: A comma-separated list of the topic names, such as orders,events,user-activity.

• Kafka format: The serialized format of the messages on the topics.

  • string (default)
  • json
  • avro

• Data includes schema definition? (only when Kafka format is json; default: off, advanced setting): When off, Streamkap infers the schema from your data, depending on the destination. When on, Streamkap expects each message key and value to contain schema and payload structures.

• Upstream records already carry Streamkap metadata (default: off, advanced setting): Turn on when the topics are fed by another Streamkap pipeline and records already carry _streamkap_offset, _streamkap_ts_ms and _streamkap_source_ts_ms, so those upstream values are kept.

Click Save.

Adding Topics Programmatically

You can set the topics of an existing Kafka Direct source via the API or Terraform using the topic.include.list.user.defined parameter (Terraform: topic_include_list_user_defined).

This is the same list as the Topics field in the UI: setting it replaces the source’s topic list.

Troubleshooting