​

Overview

​

Prerequisites

• An external service or application capable of sending HTTP POST, PUT, or DELETE requests
• Understanding of your data format (JSON or string)
• Plan for topic naming and routing strategy
• API key for secure access to your webhook endpoint

​

How It Works

1. Webhook Generation: Once created, the connector generates a unique webhook URL and API key.
2. Flexible Routing: Requests to https://webhook.endpoint/topic_name are routed to the topic_name topic. Requests to the base URL route to the default topic.
3. Automatic Topic Creation: If you POST to a new path (e.g., /orders), the connector can automatically create that topic, but you should pre-register it in the Schema section for proper configuration.
4. Data Transformation: Incoming data is converted to JSON format and optionally enriched with metadata (e.g., HTTP method, deletion markers).
5. Message Keys: The connector extracts Kafka message keys from HTTP headers, enabling downstream systems to perform upserts and deletes based on unique identifiers.

​

Streamkap Setup (UI)

1. Navigate to Sources and choose Webhook.
2. Fill in the fields:
   1. Name – A memorable identifier for this webhook source (e.g., “External Service Webhook”).
   2. Webhook URL – Auto-generated and displayed as read-only after creation. This URL is unique to your webhook. Share this URL with external services. Example: https://wh-abc123xyz-tenant-internal.streamkap.net.
   3. API Key – Auto-generated and displayed as read-only after creation. External services must include this in requests using the X-API-Key header.
   4. Data Format – Choose json or string:
      • JSON – Expects JSON payloads; automatically parsed and converted to Kafka records.
      • String – Treats the raw HTTP body as a string value.
   5. Add Delete Field – Enable to add a delete: true field when DELETE HTTP requests are received. Useful for marking deletions in downstream systems.
   6. Camel Message Header Key – The name of the HTTP header containing the Kafka message key (default: key). This is critical for enabling upserts and deletes in downstream consumers.
      • The header value must be a valid JSON object representing the key fields.
      • The same key field(s) must also be present in the request body/payload.
      • Example: If you want to upsert records by id, set this to key and include both the header and body field.
      • See Message Key Extraction section below for detailed examples.
3. Schema Configuration:
   1. Define the topics where incoming data will be stored.
   2. Default Topic Name – The topic that receives all requests to the base webhook URL (default: default_topic). Must be added to the Schema section below.
   3. Topic Names – Additional topics to route requests to based on URL paths. Add topic names you plan to use. Examples:
      • orders – Requests to https://wh-abc123xyz-tenant-internal.streamkap.net/orders route here.
      • payments – Requests to https://wh-abc123xyz-tenant-internal.streamkap.net/payments route here.
      • events – Requests to https://wh-abc123xyz-tenant-internal.streamkap.net/events route here.
   4. Important: Always include default_topic in the list. If you expect dynamic topics, pre-register them here or wait for the automatic topic discovery service to detect them (which may take time).
4. Save the Source.
5. Attach to Pipelines and configure downstream Sinks as needed.

​

Authentication

X-API-Key: <YOUR_API_KEY>

​

Example Request with API Key

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/orders \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -d '{"order_id": 12345, "customer": "John Doe", "amount": 99.99}'

​

Message Key Extraction

​

How It Works

1. Header Configuration: Set the Camel Message Header Key to the name of the HTTP header containing the key (default: key).
2. Key Format: The header value must be a valid JSON object representing the key fields.
3. Dual Presence: The key field(s) must appear in both the HTTP header AND the request body for consistency.
4. Single Record Assumption: The connector assumes each HTTP request contains exactly one message with one key.

​

Key Extraction Examples

​

Example 1: Simple ID-Based Key

• Camel Message Header Key: key
• Body includes id field

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/orders \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "key: {\"id\": 12345}" \
  -d '{
    "id": 12345,
    "customer": "John Doe",
    "amount": 99.99,
    "timestamp": "2024-01-15T10:30:00Z"
  }'

• Message Key: {"id": 12345}
• Message Value:

{
  "id": 12345,
  "customer": "John Doe",
  "amount": 99.99,
  "timestamp": "2024-01-15T10:30:00Z"
}

​

Example 2: Composite Key

• Camel Message Header Key: key
• Body includes both customer_id and order_number fields

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/orders \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "key: {\"customer_id\": \"CUST-123\", \"order_number\": \"ORD-001\"}" \
  -d '{
    "customer_id": "CUST-123",
    "order_number": "ORD-001",
    "amount": 150.50,
    "status": "pending"
  }'

• Message Key: {"customer_id": "CUST-123", "order_number": "ORD-001"}
• Message Value:

{
  "customer_id": "CUST-123",
  "order_number": "ORD-001",
  "amount": 150.50,
  "status": "pending"
}

​

Example 3: DELETE with Key

• Camel Message Header Key: key
• Add Delete Field: Enabled

curl -X DELETE https://wh-abc123xyz-tenant-internal.streamkap.net/users \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "key: {\"user_id\": \"USER-456\"}" \
  -d '{
    "user_id": "USER-456"
  }'

• Message Key: {"user_id": "USER-456"}
• Message Value:

{
  "user_id": "USER-456",
  "delete": true
}

​

Example 4: Custom Header Name

• Camel Message Header Key: msg_key (custom header name)
• Body includes product_id field

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/products \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "msg_key: {\"product_id\": \"PROD-789\"}" \
  -d '{
    "product_id": "PROD-789",
    "name": "Laptop",
    "price": 999.99,
    "stock": 50
  }'

• Message Key: {"product_id": "PROD-789"}
• Message Value:

{
  "product_id": "PROD-789",
  "name": "Laptop",
  "price": 999.99,
  "stock": 50
}

​

Best Practices for Message Keys

• Always Include Both: Ensure the key field(s) exist in both the HTTP header and the request body.
• Use Immutable IDs: Use unique, immutable identifiers (e.g., order ID, user ID) as keys to avoid data inconsistencies.
• Composite Keys: For datasets requiring uniqueness across multiple fields, use composite keys (e.g., customer_id + order_number).
• Consistency Across Calls: Use the same key structure for all requests to the same topic.
• Valid JSON: The header value must be valid JSON. Invalid JSON in the key header will cause request failures.

​

Using Your Webhook

• Webhook URL: The endpoint external services should POST/PUT/DELETE to
• API Key: Required for authentication via X-API-Key header

​

Routing Behavior

​

Data Format & Transformation

​

JSON Format

{
  "order_id": 12345,
  "customer": "John Doe",
  "amount": 99.99
}

​

String Format

"raw text data"

{
  "data": "raw text data"
}

​

DELETE Method Handling

{
  "order_id": 12345,
  "delete": true
}

​

Topic Discovery & Auto-Creation

• Pre-registered Topics – Topics explicitly added in the Schema section always accept requests.
• Dynamic Topics – New paths automatically create topics, but:
  • The topic must be added to the Schema section in advance for guaranteed routing.
  • If not pre-registered, a background topic discovery service will detect and register new topics, but this may take time (typically a few minutes).
  • To avoid delays, register all expected topics upfront.

​

Example Workflow

1. You plan to receive data for orders, payments, and events.
2. Add all three topics to the Topic Names field in the Schema section.
3. External services can now reliably POST to /orders, /payments, or /events.
4. If a request arrives for /refunds (not pre-registered), the connector will create the topic, but there may be a delay before it appears in your pipeline.

​

Request Limits & Performance

​

Data Format Examples

​

JSON POST to Orders Topic with Key

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/orders \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "key: {\"order_id\": \"ORD-001\"}" \
  -d '{
    "order_id": "ORD-001",
    "customer_id": "CUST-123",
    "amount": 150.50,
    "timestamp": "2024-01-15T10:30:00Z"
  }'

• Key: {"order_id": "ORD-001"}
• Value:

{
  "order_id": "ORD-001",
  "customer_id": "CUST-123",
  "amount": 150.50,
  "timestamp": "2024-01-15T10:30:00Z"
}

​

String POST to Events Topic

curl -X POST https://wh-abc123xyz-tenant-internal.streamkap.net/events \
  -H "Content-Type: text/plain" \
  -H "X-API-Key: sk-1234567890abcdef" \
  -d 'User logged in from IP 192.168.1.100'

{
  "data": "User logged in from IP 192.168.1.100"
}

​

DELETE Request with Delete Field and Key

curl -X DELETE https://wh-abc123xyz-tenant-internal.streamkap.net/users \
  -H "X-API-Key: sk-1234567890abcdef" \
  -H "key: {\"user_id\": \"USER-456\"}" \
  -d '{"user_id": "USER-456"}'

• Key: {"user_id": "USER-456"}
• Value:

{
  "user_id": "USER-456",
  "delete": true
}

​

Common Scenarios

​

Scenario 1: Multi-Topic Event Ingestion with Upserts

• Default Topic: default_topic
• Topic Names: orders, payments, events, notifications
• Data Format: JSON
• Camel Message Header Key: key

• Third-party service POSTs orders to .../orders with a key header containing the order ID.
• Billing system POSTs payments to .../payments with a key header containing the payment ID.
• Downstream sink connectors upsert records based on these keys, updating existing records instead of creating duplicates.
• Each goes to its respective topic, enabling independent processing.

​

Scenario 2: Generic Event Logging

• Default Topic: webhook_events
• Topic Names: (empty or just default)
• Data Format: String

• All requests regardless of path go to webhook_events.
• Useful for centralized logging of third-party webhooks.

​

Scenario 3: Soft Deletes with Tombstones

• Default Topic: users
• Add Delete Field: Enabled
• Data Format: JSON
• Camel Message Header Key: key

• POST {"user_id": 123, "name": "Alice"} with header key: {"user_id": 123} creates a record.
• DELETE with {"user_id": 123} and header key: {"user_id": 123} adds "delete": true to mark deletion.
• Downstream systems can process deletes accordingly, removing records by key.

​

Integration Examples

​

Integrating Shopify Webhooks

Webhook URL: https://wh-abc123xyz-tenant-internal.streamkap.net/shopify_orders
Header X-API-Key: sk-1234567890abcdef

​

Integrating GitHub Webhooks

Webhook URL: https://wh-abc123xyz-tenant-internal.streamkap.net/github_events
Header X-API-Key: sk-1234567890abcdef

​

Custom Application Integration with Keys

import requests
import json

API_KEY = "sk-1234567890abcdef"
WEBHOOK_URL = "https://wh-abc123xyz-tenant-internal.streamkap.net/my_events"

headers = {
    "X-API-Key": API_KEY,
    "Content-Type": "application/json",
    "key": json.dumps({"event_id": "EVT-001"})
}

data = {
    "event_id": "EVT-001",
    "event_type": "purchase",
    "user_id": "USER-001",
    "amount": 99.99
}

response = requests.post(WEBHOOK_URL, json=data, headers=headers)
print(response.status_code)  # 202 Accepted

​

Error Handling

​

Security Notes

• API Key Protection: Treat the API key like a password. Do not commit it to version control or expose in client-side code. Store securely in environment variables or secrets management systems.
• HTTPS Only: Always use HTTPS for webhook requests in production. Plain HTTP is not recommended.
• Key Header Security: The key header contains sensitive identifying information. Ensure HTTPS is used to prevent interception.
• IP Whitelisting: If available, restrict webhook access to known IP ranges of external services.
• Payload Validation: Implement server-side validation in downstream consumers to ensure data integrity and key consistency.
• Rate Limiting: External services should implement exponential backoff for 429 responses.

​

Limitations

• No Built-in Deduplication: If external services retry requests, duplicate records may appear. Implement deduplication downstream using unique message keys or IDs.
• Single Record per Request: The connector assumes each HTTP request contains exactly one Kafka message. Batch requests are not supported at the connector level.
• Single-Instance Deployment: The connector runs as a single task (tasks.max=1). Horizontal scaling requires multiple connector instances.
• No Request Buffering: Incoming requests are processed sequentially. High-volume ingestion may experience latency.
• Topic Pre-Registration Recommended: While auto-creation works, pre-registering topics in the Schema section ensures reliable routing without delays.
• Key Header Validation: Invalid JSON in the key header will cause the request to fail. Always validate key format before sending.

​

Monitoring & Troubleshooting

​

Checking Message Flow

• Events Written – Total messages ingested from the webhook
• Volume – Data volume in bytes
• Topic Distribution – Messages per topic

​

Common Issues