Redis (Generic)

Configure a generic Redis destination in Streamkap, including ACL user setup, connection settings, and key patterns for SET, HSET, and other commands.

Prerequisites

Redis version ≥ 6.0
A Redis user with sufficient privileges to write data
Network access from Streamkap to your Redis instance

Redis Setup

1. Grant Database Access

Configure one of the Connection Options to ensure Streamkap can reach your Redis instance.

2. Create Database User

It’s recommended to create a separate user for the Connector to access your Redis instance. Below is an example that creates a user with appropriate permissions.

# Create a user for Streamkap (Redis 6.0+)
# Replace {password} with a strong password
ACL SETUSER streamkap_user on >{password} ~* +@write +@read +@connection

# Verify the user was created
ACL LIST

The ACL configuration above grants:

on: Activates the user
>{password}: Sets the password
~*: Allows access to all keys
+@write: Allows all write commands
+@read: Allows all read commands
+@connection: Allows connection-related commands

3. Connection Details

You’ll need the following information for the Streamkap Setup:

Hostname/IP: The Redis server hostname or IP address
Port: Default is 6379
Database: Redis database number (0-15 by default)
Username: The username created above (if using ACL)
Password: The password for the user

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Destination

Navigate to Add Connectors.
Choose Redis.

2. Connection Settings

Name: Enter a unique and memorable name for this Connector.
Redis Type: Choose the type of Redis deployment:
- Standalone: Single Redis instance
- Cluster: Redis Cluster deployment
Hostname: Specify the hostname or IP address of your Redis server.
Port: Default is 6379.
Database: Specify the Redis database number to stream data to (0-15 by default).
- Note: Not applicable for Redis Cluster deployments.
Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your Redis instance. This is necessary if the Connector cannot connect directly.
- See SSH Tunnel for setup instructions.
Username: Username to access Redis (case sensitive). By default, Streamkap scripts use streamkap_user.
- Leave empty for Redis versions < 6.0 without ACL support.
Password: Password to access the Redis instance.
Enable TLS?: Whether to use an encrypted TLS/SSL connection to the Redis server. Default is Yes.
Allow Insecure Connections?: Allow connections with invalid or self-signed certificates. Default is No.
- Not recommended for production environments.

3. Data Format Settings

Redis Command: The Redis command to use for writing data. Options include:
- SET: Simple key-value pairs (default)
- HSET: Hash maps
- LPUSH: List (left push)
- RPUSH: List (right push)
- SADD: Set
- ZADD: Sorted set
- XADD: Redis Stream
Key Pattern: Pattern for generating Redis keys from Kafka records. Supports placeholders:
- ${topic}: The Kafka topic name
- ${key}: The Kafka message key
- Default: ${topic}:${key}
Time to Live (TTL): Optional. Set a TTL in seconds for Redis keys. Leave empty for no expiration.

4. Advanced Parameters

Batch Size: Number of records to write in a single batch. Default is 1000.
- Adjust based on your Redis instance’s performance.
Enable Multi-Exec: Whether to use Redis MULTI/EXEC transactions for batch writes. Default is No.
- Improves atomicity but may impact throughput.
Character Encoding: Character set for encoding string values. Default is UTF-8.

Click Save.

Data Mapping

The connector maps Kafka records to Redis based on the selected command:

SET Command

Key: Generated from the key pattern
Value: The entire Kafka record value (JSON string or serialized format)

HSET Command

Key: Generated from the key pattern
Fields: Each field in the Kafka record becomes a hash field
Values: Corresponding field values

XADD Command (Redis Streams)

Stream Name: Generated from the key pattern
Fields: Kafka record fields become stream entry fields

Collection Commands (LPUSH, RPUSH, SADD, ZADD)

Key: Generated from the key pattern
Elements: Values extracted from Kafka records

Troubleshooting

Connection failures

If you cannot connect to Redis:

Verify the hostname, port, and network connectivity
Check firewall rules allow traffic on the Redis port
Ensure Redis is configured to accept connections from Streamkap’s IP addresses
For Redis Cluster, verify all cluster nodes are accessible
Check TLS/SSL settings match your Redis configuration

Authentication errors

If you’re experiencing authentication issues:

Verify the username and password are correct (usernames are case-sensitive)
For Redis 6.0+, ensure the user has appropriate ACL permissions
Check if requirepass is set correctly in redis.conf for Redis versions < 6.0
Ensure the user has write permissions to the target database

Performance issues

If you’re experiencing slow write performance:

Increase the batch size to write more records per batch
Enable Multi-Exec for better transaction handling
Monitor Redis metrics (CPU, memory, network I/O)
Consider using Redis Cluster for horizontal scaling
Check network latency between Streamkap and Redis

Key conflicts or overwrites

If you’re experiencing unexpected key overwrites:

Review your key pattern to ensure uniqueness
Consider including additional fields in the key pattern (e.g., timestamp)
For append operations, use LPUSH/RPUSH instead of SET
Enable TTL to automatically expire old keys
Review your data model and key naming strategy

Data type mismatches

If you encounter errors related to data types:

Ensure the Redis command matches your data structure
Verify the Kafka record format is compatible with the selected command
Check that nested structures are properly handled for HSET commands
Review the character encoding settings for string data
Validate that sorted set scores are numeric for ZADD commands