Skip to main content
Kafka topics serve as the central data channels in your Streamkap service. The Topics page provides visibility into topic metrics, message inspection capabilities, and partition configuration management.

​
Overview

The Topics page displays aggregate metrics and a searchable list of all Kafka topics in your service:

​
Metrics Overview

Four key metric cards provide at-a-glance insights:
  • Topics: Total count of Kafka topics in the service (including dead letter queues)
  • Volume: Total data volume processed across all topics
  • Errors: Count of error messages sent to dead letter queues
  • Events: Total number of events/messages processed

​
Time Series Chart

The interactive time series chart displays topic metrics over selectable time ranges:
  • Time Range Options: Last 24 hours, Last 7 days, Last 30 days
  • Metric Selection: Toggle between Volume, Errors, and Events
  • Data Granularity: Hourly data points for 24-hour view, daily aggregates for longer periods
  • Hover Tooltips: View exact values at specific time points

​
Topics Table

The main table lists all topics with the following columns:
  • Name: The Kafka topic name
    • Click to navigate to topic detail page
    • Dead letter queue topics follow the pattern: [connector-name].streamkap.deadletterqueue
  • Entity: The associated connector (source or destination)
    • Shows connector icon and name
    • Displays connector type (SOURCE or DESTINATION)
    • Click connector name to navigate to connector detail page
  • Volume: Data volume for this topic
  • Errors: Error count for this topic
  • Written: Messages written to this topic
  • Last: Timestamp of last message activity
  • Actions: Three-dot menu for topic-specific actions

​
Search and Filtering

  • Search Bar: Filter topics by name using the search input
  • Refresh Button: Manually refresh topic metrics and list
  • Pagination: Navigate through topics with configurable page size (10, 20, 50, or 100 per page)

​
Topic Detail Page

Click any topic name to view detailed information across three tabs: Status, Metadata, and Settings.

​
Status Tab

The Status tab displays topic-specific metrics and message browser:
Metric Cards:
  • Written: Number of messages written to the topic
  • Volume: Total data volume in bytes
  • Errors: Error count for this topic
  • Events: Total event count
Message Browser: Browse and inspect individual messages within the topic:
  • Filter Messages: Filter messages by partition, offset, or other criteria
  • Refresh Button: Reload messages from Kafka
  • Partition Indicator: Shows number of partitions (e.g., “1 partition(s)”)
  • Sort Order: Dropdown to toggle between “Oldest First” and “Newest First”
  • Message Table Columns:
    • Offset: Message offset within the partition
    • Partition: Partition number
    • Timestamp: When the message was written
    • Key: Message key (if present)
    • Value: Message payload (click to expand and view)
Empty State: Topics with no messages display “No matching records found” with a search icon.
Dead letter queue topics (.streamkap.deadletterqueue) are typically empty unless messages have failed processing and been sent to the DLQ.

​
Metadata Tab

The Metadata tab displays comprehensive Kafka topic configuration and health information:
Health Status Monitor the overall health and replication status of your topic:
  • Status: Overall topic health (Healthy, Degraded, or Unhealthy)
  • Total Replicas: Total number of replicas across all partitions
  • In-Sync Replicas: Number of replicas that are caught up with the leader
  • Under-Replicated: Number of replicas that are lagging behind the leader
  • Offline Partitions: Number of partitions without an active leader
A healthy topic should have all replicas in-sync (In-Sync Replicas = Total Replicas) and zero under-replicated or offline partitions.
Partition Details View partition-level configuration and status:
  • Total Partitions: Number of partitions in the topic
  • Replication Factor: Number of replicas for each partition
  • Under-Replicated: Count of under-replicated partitions
  • Offline Count: Count of offline partitions
Individual Partitions Table: Each partition is displayed with:
  • Partition ID: Unique partition identifier
  • Replicas: List of broker IDs hosting replicas
  • In-Sync Replicas: List of broker IDs with synchronized replicas
  • Status: Per-partition health status
Retention Configuration Control how long Kafka retains messages:
  • Retention Time: Time-based retention (e.g., 7d for 7 days, -1 for unlimited)
  • Retention Bytes: Size-based retention per partition (in bytes, -1 for unlimited)
  • Delete Retention: Time to retain delete markers for log compaction
  • Cleanup Policy: Message cleanup strategy
    • delete: Delete old messages based on retention settings
    • compact: Keep only the latest value for each key
    • delete,compact: Apply both strategies
For CDC topics, a retention time of 7 days is typically sufficient for most use cases. Adjust based on your downstream consumer lag tolerance and recovery requirements.
Message & Compression Configuration View message handling and compression:
  • Max Message Size: Maximum size of a single message (default: 64 MB)
  • Compression Type: Compression algorithm applied to messages
    • lz4: Fast compression with good ratio (recommended for CDC)
    • gzip: Higher compression ratio, more CPU intensive
    • snappy: Balanced compression and speed
    • zstd: Modern compression with excellent ratio and speed
    • none: No compression
  • Min In-Sync Replicas: Minimum replicas that must acknowledge writes for success
  • Timestamp Type: Source of message timestamps
    • CreateTime: Use timestamp from producer
    • LogAppendTime: Use timestamp when message is written to Kafka
Setting Min In-Sync Replicas too high can cause write failures if brokers are unavailable. A value of 2 balances durability and availability for a replication factor of 3.
Segment Configuration View Kafka log segment parameters:
  • Segment Time: Maximum time before creating a new log segment (e.g., 1h for 1 hour)
  • Segment Size: Maximum size of a log segment before rolling to a new segment (e.g., 1.02K MB)
Smaller segments enable finer-grained retention and compaction but increase overhead. For high-throughput CDC topics, segment times of 1 hour and sizes around 1 GB are typical.
Other Configuration Additional Kafka topic settings:
  • Unclean Leader Election: Whether to allow out-of-sync replicas to become leaders
    • true: Prioritize availability (risk of data loss)
    • false: Prioritize durability (recommended for CDC)

​
Settings Tab

The Settings tab provides partition configuration management:
Partition Count Configuration: Adjust the number of partitions for horizontal scaling:
  • Slider Control: Drag to increase partition count
  • Current Value Display: Shows the active partition count
  • Minimum Value: Cannot be lower than current count (Kafka limitation)
  • Save Button: Apply partition changes
Increasing partitions will cause messages to be read out of order and may reorganize destination tables.This is a critical operation that requires careful planning. See the safe procedure below.

​
Dead Letter Queue Topics

Streamkap automatically creates a dead letter queue (DLQ) topic for each destination connector: Naming Convention: [connector-name].streamkap.deadletterqueue Purpose:
  • Captures messages that fail processing at the destination
  • Prevents pipeline stoppage due to individual message failures
  • Enables inspection and recovery of failed messages
Common Failure Causes:
  • Schema mismatches between source and destination
  • Data type conversion errors
  • Constraint violations in destination tables
  • Malformed message payloads
Monitoring:
  • Check the Errors metric on the main Topics page
  • Inspect DLQ topic messages in the Status tab
  • Set up alerts for DLQ activity (see Alerts)
If messages appear in a DLQ topic, investigate the pipeline logs and destination connector settings to identify the root cause. Fix the issue, then consider replaying the failed messages if needed.

​
Partition Management

​
When to Increase Partitions

Consider increasing partitions when:
  • Throughput bottlenecks: Single partition cannot handle message volume
  • Parallelism needed: Want multiple consumers to process messages concurrently
  • Scaling destinations: Destination can benefit from parallel writes

​
Safe Partition Increase Procedure

Follow this 8-step procedure to safely increase partitions:
1

Stop sources/producers

Pause all source connectors writing to this topic to prevent new messages.
2

Wait for consumer lag to drop to 0

Ensure all in-flight messages have been consumed and processed.
3

Stop destinations streaming this topic

Pause destination connectors consuming from this topic.
4

Truncate destination table (Optional - depending on destination)

For most destinations, truncate the target table to prepare for potential re-ingestion.Skip this step if using ClickHouse or Snowflake (see below).
5

Increase topic partitions

Use the Settings tab slider to increase the partition count and click Save.
6

Start sources/producers

Resume source connectors to begin writing messages to the new partition layout.
7

Trigger snapshot (Optional - depending on destination)

For some destinations, trigger a snapshot to re-sync data.Skip this step if using ClickHouse or Snowflake (see below).
8

Start destinations

Resume destination connectors to consume messages from all partitions.

​
Destination-Specific Handling

ClickHouse:
  • Uses ReplacingMergeTree engine with background merging
  • Handles out-of-order records automatically through de-duplication
  • Does not require table truncation or snapshot triggering
Snowflake:
  • Uses Dynamic Tables or dbt with sorting by _streamkap_ts_ms DESC, _streamkap_offset DESC
  • Post-ingestion de-duplication handles out-of-order records
  • Does not require table truncation or snapshot triggering
Other Destinations:
  • BigQuery, Databricks, Redshift, etc.
  • Require the full 8-step procedure to maintain data consistency
Failing to follow the safe partition increase procedure can result in:
  • Duplicate records in destination tables
  • Out-of-order data processing
  • Inconsistent table state
  • Data quality issues

​
Performance Tuning

If you are experiencing performance issues or lag, consider the following adjustments:

​
Increase Maximum Poll Records

In your Connector’s Settings under Advanced, set the Maximum poll records to a higher value such as 25000, 50000, or 80000—depending on your record sizes. This allows the Destination to fetch more records per poll and can improve throughput.

​
Increase Topic Partitions

On the Topics page, increase the number of partitions for your topic to at least 5. More partitions enable greater parallelism and can help distribute the load more efficiently.
Increasing partitions won’t affect existing data but will only apply to new data written to the topic. High lag may persist on the existing partitions until they are fully processed.

​
Increase Tasks

In your Connector’s Settings, set the Tasks to a higher value to allow more tasks to process topic partitions in parallel. This can help reduce lag and improve processing speed.

​
Best Practices

  1. Monitor DLQ Topics: Set up alerts for dead letter queue activity to catch processing failures quickly
  2. Plan Partition Changes: Schedule partition increases during maintenance windows when data pipelines can be paused
  3. Use Appropriate Partition Counts: Start with fewer partitions and scale up based on throughput needs (over-partitioning can reduce performance)
  4. Inspect Failed Messages: Regularly review DLQ topics to identify recurring issues
  5. Document Partition Changes: Record partition increase operations and their impact on downstream systems
  6. Test Before Scaling: If possible, test partition increases in a non-production environment first
  7. Monitor Topic Metrics: Regularly check topic metrics to detect lag or errors early
  8. Adjust Incrementally: Make performance tuning changes one at a time and monitor the impact

​
Troubleshooting

​
Topic Not Appearing in List

If a topic is missing from the Topics page:
  1. Check Connector Status: Ensure the source or destination connector is running
  2. Verify Topic Creation: Some topics are created only after first message write
  3. Refresh the Page: Click the refresh button to reload the topic list
  4. Check Kafka: Use Kafka Access to verify the topic exists in Kafka directly

​
High Error Count

If a topic shows a high error count:
  1. Inspect DLQ Messages: Navigate to the dead letter queue topic and review failed messages
  2. Check Pipeline Logs: Review logs for the associated pipeline (Logs)
  3. Review Destination Settings: Verify destination schema and configuration
  4. Check for Alerts: Review alerts for this topic and associated connectors (Alerts)
  5. Validate Schema: Ensure source schema matches destination expectations

​
Partition Increase Not Taking Effect

If partition changes don’t apply:
  1. Check Save Button: Ensure you clicked Save after adjusting the slider
  2. Verify Kafka State: Use Kafka Access to confirm partition count in Kafka
  3. Review Logs: Check service logs for partition update errors
  4. Kafka Limitations: Remember that partition count can only increase, never decrease
  5. Wait for Propagation: Some changes may take a few moments to reflect in the UI

​
Messages Not Appearing in Browser

If the message browser shows “No matching records found”:
  1. Check Time Range: Ensure messages exist in the displayed time range
  2. Verify Message Production: Confirm the source is actively writing messages
  3. Check Partition Filter: Remove any partition filters that may be hiding messages
  4. Refresh Data: Click the refresh button to reload messages from Kafka
  5. Inspect Kafka Directly: Use Kafka Access to verify messages exist in the topic
  • Pipelines - Configure data pipelines that read from and write to topics
  • Sources - Source connectors that write to topics
  • Destinations - Destination connectors that read from topics
  • Kafka Access - Direct access to Kafka for advanced topic management
  • Alerts - Set up alerts for topic errors and DLQ activity
  • Logs - Troubleshoot topic and message processing issues
⌘I