In Streamkap, a Topic is like a database table where data flows between sources and destinations. Topics organize and separate different streams of data, making it easy to manage, monitor, and scale your data pipelines. Each topic can be configured for performance, reliability, and scalability. This page explains how to monitor, tune, and safely modify topic settings.

​
Monitoring Topics

On the Topics page, you can:
  • View all topics, their volume, error counts, and event counts at a glance
  • Drill into a topic to see message offsets, partitions, and recent activity
  • Search for topics by name
Topics Overview Selecting a topic shows detailed stats, including written events, volume, errors, and a message browser: Topic Details

​
Performance Tuning

If you are experiencing performance issues or lag, consider the following adjustments:
  • Increase maximum poll records
    In your Connector’s Settings under Advanced, set the Maximum poll records to a higher value such as 25000, 50000, or 80000—depending on your record sizes. This allows the Destination to fetch more records per poll and can improve throughput.
  • Increase topic partitions
    Data consistencyIncreasing the number of partitions for a topic can improve throughput and parallelism, but it must be done carefully to avoid data consistency issues.For most destinations, follow the safe procedure below. However, ClickHouse (via ReplacingMergeTree background merging) and Snowflake (using Dynamic Tables/dbt, sorted by _streamkap_ts_ms DESC, _streamkap_offset DESC) can safely handle out-of-order records through post-ingestion de-duplication.Safe Partition Increase Procedure:
    1. Stop sources/producers
    2. Wait for consumer lag to drop to 0
    3. Stop destinations streaming this topic
    4. Truncate destination table (Optional - depending on destination)
    5. Increase topic partitions
    6. Start sources/producers
    7. Trigger snapshot (Optional - depending on destination)
    8. Start destinations
    Partition Settings
    On the Topics page, increase the number of partitions for your topic to at least 5. More partitions enable greater parallelism and can help distribute the load more efficiently.
    Increasing partitions won’t affect existing data but will only apply to new data written to the topic. So high lag may persist on the existing partitions until they are fully processed.
  • Increase tasks
    In your Connector’s Settings, set the Tasks to a higher value to allow more tasks to process topic partitions in parallel. This can help reduce lag and improve processing speed.

​
Best Practices

  • Monitor topic metrics regularly to detect lag or errors early.
  • Adjust partitions and tasks incrementally, monitoring the impact after each change.
  • Always follow the safe procedure when increasing partitions, especially for destinations that do not support out-of-order de-duplication.