Kafka (Reading)
Read directly from Kafka
Streamkap supports reading from Kafka.
Customers on a paid plan can enable direct access via Proxy.
Connection details user and topic prefix must be configured for each proxy instance:
- Kafka brokers example:
<tenant-name>.streamkap.net:32400
,<tenant-name>.streamkap.net:32401
,<tenant-name>.streamkap.net:32402
- Security protocol:
SASL_SSL
- SASL mechanism: PLAIN
- username/password
- topic prefixes and read access permissions
CLI example:
kcat -b some-tenant.streamkap.net:32430 -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN -X sasl.username=some_user -X sasl.password=***** -C -t my-topic -K \| -c1 -o -1
Confluent's Kafka Python Client example:
from confluent_kafka import Consumer
conf = {
'bootstrap.servers': 'my-tenant.streamkap.net:32400,my-tenant.streamkap.net:32401,my-tenant.streamkap.net:32402',
'error_cb': error_callback,
'security.protocol': 'SASL_SSL',
'sasl.mechanism': 'PLAIN',
'sasl.username': 'myuser',
'sasl.password': os.environ['STREAMKAP_USER_PASSWORD'],
'client.id': socket.gethostname(),
# Required to trust AWS root certifiates. Without this we will
# not be able to successfully coplete the SSL handshake.
'ssl.ca.location': certifi.where(),
}
c = Consumer(conf)
Integrations supported via Kafka direct read access:
- Materialize: link to docs
- Tinybird: link to docs
- SingleStore: link to docs
- Starburst: link to docs
- StarTree: link to docs
- Microsoft Fabric: link to docs
Updated 14 days ago