Kafka (Push)

Replicate topics from Streamkap to another Kafka cluster using Avro or JSON format, with configurable topic name prefixes and suffixes.

The Kafka (Push) Destination allows replication of topics from Streamkap to another Kafka cluster.

Prerequisites

• Provide Streamkap’s services access to the Kafka cluster using one of the available Connection Options
• Kafka bootstrap server URLs (external not internal listener addresses) e.g. broker-1.mycompany.com:9094,broker-2.mycompany.com:9094, ...
• The topic data format. For Avro, you will need the URL for a Schema Registry service running and accessible to Streamkap e.g. https://schema-registry.mycompany.com:8081

Streamkap Setup

1. Go to Destinations and choose Kafka (Push)

2. Input the following information:

3. Name - A unique and memorable name for this Connector

4. Kafka Bootstrap Servers - A comma-separated list of host:port pairs for the destination Kafka brokers, such as broker1.kafka.company.com:9092,broker2.kafka.company.com:9092

5. Format (default: JSON) - The topic data format

   1. Avro: Messages will be serialised in Avro format using schemas from the Schema Registry
      1. Schema Registry URL - The URL for the destination Kafka Schema Registry e.g. http://schema-registry.company.com:8081
   2. JSON: Messages will be serialised in JSON format
      1. Include JSON Schema - Choose to include or exclude the schema in the JSON formatted messages

6. Topic Prefix - Specify a prefix to add to each destination topic name. This can help with organising topics in the Kafka cluster

7. Topic Suffix (Optional) - Specify a suffix to add to each destination topic name

8. Click Save