Topics

Kafka topics serve as the central data channels in your Streamkap service. The Topics page provides visibility into topic metrics, message inspection capabilities, and partition configuration management.

​

Overview

The Topics page displays aggregate metrics and a searchable list of all Kafka topics in your service:

​

Metrics Overview

Four key metric cards provide at-a-glance insights:

• Topics: Total count of Kafka topics in the service (including dead letter queues)
• Volume: Total data volume processed across all topics
• Errors: Count of error messages sent to dead letter queues
• Events: Total number of events/messages processed

​

Time Series Chart

The interactive time series chart displays topic metrics over selectable time ranges:

• Time Range Options: Last 24 hours, Last 7 days, Last 30 days
• Metric Selection: Toggle between Volume, Errors, and Events
• Data Granularity: Hourly data points for 24-hour view, daily aggregates for longer periods
• Hover Tooltips: View exact values at specific time points

​

Topics Table

The main table lists all topics with the following columns:

• Name: The Kafka topic name
  • Click to navigate to topic detail page
  • Dead letter queue topics follow the pattern: [connector-name].streamkap.deadletterqueue
• Entity: The associated connector (source or destination)
  • Shows connector icon and name
  • Displays connector type (SOURCE or DESTINATION)
  • Click connector name to navigate to connector detail page
• Volume: Data volume for this topic
• Errors: Error count for this topic
• Written: Messages written to this topic
• Last: Timestamp of last message activity
• Actions: Three-dot menu for topic-specific actions

​

Search and Filtering

• Search Bar: Filter topics by name using the search input
• Refresh Button: Manually refresh topic metrics and list
• Pagination: Navigate through topics with configurable page size (10, 20, 50, or 100 per page)

​

Topic Detail Page

Click any topic name to view detailed information across three tabs: Status, Metadata, and Settings.

​

Status Tab

The Status tab displays topic-specific metrics and message browser: Metric Cards:

• Written: Number of messages written to the topic
• Volume: Total data volume in bytes
• Errors: Error count for this topic
• Events: Total event count

Message Browser: Browse and inspect individual messages within the topic:

• Filter Messages: Apply seek or regex filters to navigate and search messages (see Filtering Messages)
• Refresh Button: Reload messages from Kafka
• Partition Indicator: Shows number of partitions (e.g., “1 partition(s)”)
• Sort Order: Dropdown to toggle between “Oldest First” and “Newest First”
• Message Table Columns:
  • Offset: Message offset within the partition
  • Partition: Partition number
  • Timestamp: When the message was written
  • Key: Message key (if present)
  • Value: Message payload (click to expand and view)

Empty State: Topics with no messages display “No matching records found” with a search icon.

​

Filtering Messages

The Filter Messages feature allows you to navigate to specific positions in the topic or search for messages matching patterns. Click the Filter Messages button to open the filter dialog. The filter dialog provides two types of filters that can be used independently:

​

Seek Filter

The Seek Filter allows you to jump to a specific position in the topic: Offset Seek:

• Select Offset from the dropdown
• Enter a numeric offset value to jump to that specific message position
• Useful for investigating specific message sequences or resuming from a known position

Timestamp Seek:

• Select Timestamp from the dropdown
• Choose a date and time using the date picker
• Jumps to the first message at or after the selected timestamp
• Useful for investigating messages from a specific time period

​

Regex Filter

The Regex Filter allows you to search for messages matching a regular expression pattern: Search In: Select which message fields to search:

• key: Search in message keys only
• value: Search in message values only
• Both (Select All): Search in both keys and values

Regex Pattern: Enter a regular expression pattern to match:

• Supports standard regex syntax (e.g., .*@gmail\.com for Gmail addresses)
• Case-sensitive by default
• Patterns are matched against the entire key or value (JSON strings)

​

Applying and Managing Filters

Apply Filter:

1. Configure your desired Seek Filter and/or Regex Filter
2. Click Apply to activate the filter
3. A “Filter applied” badge appears with filter management buttons

Remove Filter:

• Click the X button next to “Filter applied” to clear all active filters
• Click the info icon (ℹ) to see filter details
• Removing the filter returns to the default view (first messages in the topic)

Filter Combinations:

• Seek Only: Navigate to a position without filtering messages
• Regex Only: Search from the beginning of the topic
• Seek + Regex: Navigate to a position, then search from there (recommended for large topics)

​

Metadata Tab

The Metadata tab displays comprehensive Kafka topic configuration and health information: Health Status Monitor the overall health and replication status of your topic:

• Status: Overall topic health (Healthy, Degraded, or Unhealthy)
• Total Replicas: Total number of replicas across all partitions
• In-Sync Replicas: Number of replicas that are caught up with the leader
• Under-Replicated: Number of replicas that are lagging behind the leader
• Offline Partitions: Number of partitions without an active leader

Partition Details View partition-level configuration and status:

• Total Partitions: Number of partitions in the topic
• Replication Factor: Number of replicas for each partition
• Under-Replicated: Count of under-replicated partitions
• Offline Count: Count of offline partitions

Individual Partitions Table: Each partition is displayed with:

• Partition ID: Unique partition identifier
• Replicas: List of broker IDs hosting replicas
• In-Sync Replicas: List of broker IDs with synchronized replicas
• Status: Per-partition health status

Retention Configuration Control how long Kafka retains messages:

• Retention Time: Time-based retention (e.g., 7d for 7 days, -1 for unlimited)
• Retention Bytes: Size-based retention per partition (in bytes, -1 for unlimited)
• Delete Retention: Time to retain delete markers for log compaction
• Cleanup Policy: Message cleanup strategy
  • delete: Delete old messages based on retention settings
  • compact: Keep only the latest value for each key
  • delete,compact: Apply both strategies

Message & Compression Configuration View message handling and compression:

• Max Message Size: Maximum size of a single message (default: 64 MB)
• Compression Type: Compression algorithm applied to messages
  • lz4: Fast compression with good ratio (recommended for CDC)
  • gzip: Higher compression ratio, more CPU intensive
  • snappy: Balanced compression and speed
  • zstd: Modern compression with excellent ratio and speed
  • none: No compression
• Min In-Sync Replicas: Minimum replicas that must acknowledge writes for success
• Timestamp Type: Source of message timestamps
  • CreateTime: Use timestamp from producer
  • LogAppendTime: Use timestamp when message is written to Kafka

Segment Configuration View Kafka log segment parameters:

• Segment Time: Maximum time before creating a new log segment (e.g., 1h for 1 hour)
• Segment Size: Maximum size of a log segment before rolling to a new segment (e.g., 1.02K MB)

Other Configuration Additional Kafka topic settings:

• Unclean Leader Election: Whether to allow out-of-sync replicas to become leaders
  • true: Prioritize availability (risk of data loss)
  • false: Prioritize durability (recommended for CDC)

​

Settings Tab

The Settings tab provides partition configuration management: Partition Count Configuration: Adjust the number of partitions for horizontal scaling:

• Slider Control: Drag to increase partition count
• Current Value Display: Shows the active partition count
• Minimum Value: Cannot be lower than current count (Kafka limitation)
• Save Button: Apply partition changes

​

Dead Letter Queue Topics

Streamkap automatically creates a dead letter queue (DLQ) topic for each destination connector: Naming Convention: [connector-name].streamkap.deadletterqueue Purpose:

• Captures messages that fail processing at the destination
• Prevents pipeline stoppage due to individual message failures
• Enables inspection and recovery of failed messages

Common Failure Causes:

• Schema mismatches between source and destination
• Data type conversion errors
• Constraint violations in destination tables
• Malformed message payloads

Monitoring:

• Check the Errors metric on the main Topics page
• Inspect DLQ topic messages in the Status tab
• Set up alerts for DLQ activity (see Alerts)

​

Partition Management

​

When to Increase Partitions

Consider increasing partitions when:

• Throughput bottlenecks: Single partition cannot handle message volume
• Parallelism needed: Want multiple consumers to process messages concurrently
• Scaling destinations: Destination can benefit from parallel writes

​

Safe Partition Increase Procedure

Follow this 8-step procedure to safely increase partitions:

​

Destination-Specific Handling

ClickHouse:

• Uses ReplacingMergeTree engine with background merging
• Handles out-of-order records automatically through de-duplication
• Does not require table truncation or snapshot triggering

Snowflake:

• Uses Dynamic Tables or dbt with sorting by _streamkap_ts_ms DESC, _streamkap_offset DESC
• Post-ingestion de-duplication handles out-of-order records
• Does not require table truncation or snapshot triggering

Other Destinations:

• BigQuery, Databricks, Redshift, etc.
• Require the full 8-step procedure to maintain data consistency

​

Performance Tuning

If you are experiencing performance issues or lag, consider the following adjustments:

​

Increase Maximum Poll Records

In your Connector’s Settings under Advanced, set the Maximum poll records to a higher value such as 25000, 50000, or 80000—depending on your record sizes. This allows the Destination to fetch more records per poll and can improve throughput.

​

Increase Topic Partitions

On the Topics page, increase the number of partitions for your topic to at least 5. More partitions enable greater parallelism and can help distribute the load more efficiently.

​

Increase Tasks

In your Connector’s Settings, set the Tasks to a higher value to allow more tasks to process topic partitions in parallel. This can help reduce lag and improve processing speed.

​

Best Practices

1. Monitor DLQ Topics: Set up alerts for dead letter queue activity to catch processing failures quickly
2. Plan Partition Changes: Schedule partition increases during maintenance windows when data pipelines can be paused
3. Use Appropriate Partition Counts: Start with fewer partitions and scale up based on throughput needs (over-partitioning can reduce performance)
4. Inspect Failed Messages: Regularly review DLQ topics to identify recurring issues
5. Document Partition Changes: Record partition increase operations and their impact on downstream systems
6. Test Before Scaling: If possible, test partition increases in a non-production environment first
7. Monitor Topic Metrics: Regularly check topic metrics to detect lag or errors early
8. Adjust Incrementally: Make performance tuning changes one at a time and monitor the impact

​

Troubleshooting

​

Topic Not Appearing in List

If a topic is missing from the Topics page:

1. Check Connector Status: Ensure the source or destination connector is running
2. Verify Topic Creation: Some topics are created only after first message write
3. Refresh the Page: Click the refresh button to reload the topic list
4. Check Kafka: Use Kafka Access to verify the topic exists in Kafka directly

​

High Error Count

If a topic shows a high error count:

1. Inspect DLQ Messages: Navigate to the dead letter queue topic and review failed messages
2. Check Pipeline Logs: Review logs for the associated pipeline (Logs)
3. Review Destination Settings: Verify destination schema and configuration
4. Check for Alerts: Review alerts for this topic and associated connectors (Alerts)
5. Validate Schema: Ensure source schema matches destination expectations

​

Partition Increase Not Taking Effect

If partition changes don’t apply:

1. Check Save Button: Ensure you clicked Save after adjusting the slider
2. Verify Kafka State: Use Kafka Access to confirm partition count in Kafka
3. Review Logs: Check service logs for partition update errors
4. Kafka Limitations: Remember that partition count can only increase, never decrease
5. Wait for Propagation: Some changes may take a few moments to reflect in the UI

​

Messages Not Appearing in Browser

If the message browser shows “No matching records found”:

1. Check Time Range: Ensure messages exist in the displayed time range
2. Verify Message Production: Confirm the source is actively writing messages
3. Check Partition Filter: Remove any partition filters that may be hiding messages
4. Refresh Data: Click the refresh button to reload messages from Kafka
5. Inspect Kafka Directly: Use Kafka Access to verify messages exist in the topic

​

Related Documentation

• Pipelines - Configure data pipelines that read from and write to topics
• Sources - Source connectors that write to topics
• Destinations - Destination connectors that read from topics
• Kafka Access - Direct access to Kafka for advanced topic management
• Alerts - Set up alerts for topic errors and DLQ activity
• Logs - Troubleshoot topic and message processing issues