The Kafka (Push) Destination allows replication of topics from Streamkap to another Kafka cluster.

📘

Apache MirrorMaker 2

This Streamkap Destination should not be confused with Apache MirrorMaker 2. Although both replicate topics between Kafka clusters, there are fundamental differences. For example, Apache MirrorMaker 2 supports offset translation, allowing Consumers of the origin cluster to be switched over to the replica and resume without losing their position. It also supports metadata propagation i.e. topic configuration, consumer groups, ACLs.

Streamkap's Kafka Destination simply reads the messages from an origin topic and writes them to a destination topic. There is no offset translation or metadata propagation.

Prerequisites

• Provide Streamkap's services access to the Kafka cluster using one of the available Connection Options
• Kafka bootstrap server URLs (external not internal listener addresses) e.g. broker-1.mycompany.com:9094,broker-2.mycompany.com:9094, ...
• The topic data format. For Avro, you will need the URL for a Schema Registry service running and accessible to Streamkap e.g. https://schema-registry.mycompany.com:8081

Streamkap Setup

1. Go to Destinations and choose Kafka (Push)
2. Input the following information:
3. Name - A unique and memorable name for this Connector
4. Kafka Bootstrap Servers - A comma-separated list of host:port pairs for the destination Kafka brokers, such as broker1.kafka.company.com:9092,broker2.kafka.company.com:9092
5. Format (default: JSON) - The topic data format
   1. Avro: Messages will be serialised in Avro format using schemas from the Schema Registry
      1. Schema Registry URL - The URL for the destination Kafka Schema Registry e.g. http://schema-registry.company.com:8081
   2. JSON: Messages will be serialised in JSON format
      1. Include JSON Schema - Choose to include or exclude the schema in the JSON formatted messages
6. Topic Prefix - Specify a prefix to add to each destination topic name. This can help with organising topics in the Kafka cluster
7. Topic Suffix (Optional) - Specify a suffix to add to each destination topic name
8. Click Save