Skip to main content

​
Prerequisites

  • Redis version ≥ 6.0
  • A Redis user with sufficient privileges to write data
  • Network access from Streamkap to your Redis instance

​
Redis Setup

​
1. Grant Database Access

  • Configure one of the Connection Options to ensure Streamkap can reach your Redis instance.

​
2. Create Database User

It’s recommended to create a separate user for the Connector to access your Redis instance. Below is an example that creates a user with appropriate permissions. 
# Create a user for Streamkap (Redis 6.0+)
# Replace {password} with a strong password
ACL SETUSER streamkap_user on >{password} ~* +@write +@read +@connection

# Verify the user was created
ACL LIST
The ACL configuration above grants:
  • on: Activates the user
  • >{password}: Sets the password
  • ~*: Allows access to all keys
  • +@write: Allows all write commands
  • +@read: Allows all read commands
  • +@connection: Allows connection-related commands
Redis ACL compatibilityACL support requires Redis 6.0 or higher. For earlier versions, you can use the requirepass setting in redis.conf for password authentication, though this provides less granular access control.

​
3. Connection Details

You’ll need the following information for the Streamkap Setup:
  • Hostname/IP: The Redis server hostname or IP address
  • Port: Default is 6379
  • Database: Redis database number (0-15 by default)
  • Username: The username created above (if using ACL)
  • Password: The password for the user
TLS/SSL ConnectionsIf your Redis instance requires TLS/SSL, ensure you have:
  • Valid certificates
  • The correct hostname that matches the certificate
  • TLS enabled on your Redis instance

​
Streamkap Setup

Follow these steps to configure your new connector:

​
1. Create the Destination

​
2. Connection Settings

  • Name: Enter a unique and memorable name for this Connector.
  • Redis Type: Choose the type of Redis deployment:
    • Standalone: Single Redis instance
    • Cluster: Redis Cluster deployment
  • Hostname: Specify the hostname or IP address of your Redis server.
  • Port: Default is 6379.
  • Database: Specify the Redis database number to stream data to (0-15 by default).
    • Note: Not applicable for Redis Cluster deployments.
  • Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your Redis instance. This is necessary if the Connector cannot connect directly.
  • Username: Username to access Redis (case sensitive). By default, Streamkap scripts use streamkap_user.
    • Leave empty for Redis versions < 6.0 without ACL support.
  • Password: Password to access the Redis instance.
  • Enable TLS?: Whether to use an encrypted TLS/SSL connection to the Redis server. Default is Yes.
  • Allow Insecure Connections?: Allow connections with invalid or self-signed certificates. Default is No.
    • Not recommended for production environments.

​
3. Data Format Settings

  • Redis Command: The Redis command to use for writing data. Options include:
    • SET: Simple key-value pairs (default)
    • HSET: Hash maps
    • LPUSH: List (left push)
    • RPUSH: List (right push)
    • SADD: Set
    • ZADD: Sorted set
    • XADD: Redis Stream
  • Key Pattern: Pattern for generating Redis keys from Kafka records. Supports placeholders:
    • ${topic}: The Kafka topic name
    • ${key}: The Kafka message key
    • Default: ${topic}:${key}
  • Time to Live (TTL): Optional. Set a TTL in seconds for Redis keys. Leave empty for no expiration.

​
4. Advanced Parameters

  • Batch Size: Number of records to write in a single batch. Default is 1000.
    • Adjust based on your Redis instance’s performance.
  • Enable Multi-Exec: Whether to use Redis MULTI/EXEC transactions for batch writes. Default is No.
    • Improves atomicity but may impact throughput.
  • Character Encoding: Character set for encoding string values. Default is UTF-8.
Click Save.

​
Data Mapping

The connector maps Kafka records to Redis based on the selected command:

​
SET Command

  • Key: Generated from the key pattern
  • Value: The entire Kafka record value (JSON string or serialized format)

​
HSET Command

  • Key: Generated from the key pattern
  • Fields: Each field in the Kafka record becomes a hash field
  • Values: Corresponding field values

​
XADD Command (Redis Streams)

  • Stream Name: Generated from the key pattern
  • Fields: Kafka record fields become stream entry fields

​
Collection Commands (LPUSH, RPUSH, SADD, ZADD)

  • Key: Generated from the key pattern
  • Elements: Values extracted from Kafka records

​
Troubleshooting

If you cannot connect to Redis:

  • Verify the hostname, port, and network connectivity
  • Check firewall rules allow traffic on the Redis port
  • Ensure Redis is configured to accept connections from Streamkap’s IP addresses
  • For Redis Cluster, verify all cluster nodes are accessible
  • Check TLS/SSL settings match your Redis configuration
If you’re experiencing authentication issues:

  • Verify the username and password are correct (usernames are case-sensitive)
  • For Redis 6.0+, ensure the user has appropriate ACL permissions
  • Check if requirepass is set correctly in redis.conf for Redis versions < 6.0
  • Ensure the user has write permissions to the target database
If you’re experiencing slow write performance:

  • Increase the batch size to write more records per batch
  • Enable Multi-Exec for better transaction handling
  • Monitor Redis metrics (CPU, memory, network I/O)
  • Consider using Redis Cluster for horizontal scaling
  • Check network latency between Streamkap and Redis
If you’re experiencing unexpected key overwrites:

  • Review your key pattern to ensure uniqueness
  • Consider including additional fields in the key pattern (e.g., timestamp)
  • For append operations, use LPUSH/RPUSH instead of SET
  • Enable TTL to automatically expire old keys
  • Review your data model and key naming strategy
If you encounter errors related to data types:

  • Ensure the Redis command matches your data structure
  • Verify the Kafka record format is compatible with the selected command
  • Check that nested structures are properly handled for HSET commands
  • Review the character encoding settings for string data
  • Validate that sorted set scores are numeric for ZADD commands