Streaming Agents

Run LLM-powered agents on your Kafka streams — process each record with a model and tools, validate against a schema, and write results back to a topic

A streaming agent is a Streamkap resource type alongside sources, destinations, pipelines, and transforms. It reads records from a Kafka topic, runs each one through an LLM (optionally calling tools), validates the result against a schema you declare, and writes the output to another topic. Records it can’t process are routed to a dead-letter topic.

Agents are independent of sources and destinations. They bind to a Kafka topic, whichever way that topic was produced: by a source connector, a transform, another agent, or your own producer. Use them to classify, enrich, redact, or summarize streaming records in real time without hand-building a Kafka consumer and an LLM loop.

How It Works

1. An agent consumes records from one input Kafka topic.
2. For each record, it calls your chosen LLM with your system prompt and the record as input.
3. If the agent has tools, the model can call them and feed the results back to itself, looping until it produces a final answer (up to a maximum number of steps).
4. The response is validated against your optional output schema and written to the output topic.
5. Records that can’t be processed — schema mismatch, tool failure, or unparseable output — are routed to a dead-letter topic (dlq.<output-topic>).

Every LLM and tool call runs on your own credentials — you bring an LLM connection (your provider API key), and all model usage is billed by your provider. Where available, Streamkap also provides managed default connections (an LLM and a vector store) so you can try your first agent without any keys — they’re for quick tests, not production data. See Streamkap Default Connections.

Where to Find It

Streaming Agents is a project-level feature. Open Agentic (Beta) in the project sidebar to reach it. The section is organized into tabs:

Key Concepts

• Agent — a deployed job that processes one input topic. Agents come in two shapes, selected automatically from whether you add tools:
  • Workflow — no tools; a single LLM pass per record (classify, summarize, redact).
  • ReAct — one or more tools; the model reasons, calls a tool, feeds the result back, and repeats until it produces an answer.
• Connection — a saved, encrypted credential shared across every agent in your organization: an LLM provider, an external MCP server, an HTTP endpoint, or a vector store. See Connections.
• Tool — something the model can call during a run: an HTTP request, a Script (JavaScript or Python), an External MCP server, or Streamkap MCP (the agent operating your Streamkap platform).
• Knowledge Base — a streaming pipeline that embeds a Kafka topic into a vector store so agents can retrieve from live data. See Knowledge Bases.
• Memory — optional short-term (time-boxed) and long-term (vector store) context carried across records.
• Dead-letter topic (DLQ) — where unprocessable records land, derived as dlq.<output-topic>.

Permissions

Access is controlled by two permissions on your Streamkap roles:

The sidebar entry is hidden if you don’t have read:agents. Read-only users can view the screens but write actions are disabled. If you don’t see the feature or an action, ask an admin for the appropriate permission.

Limits

Next Steps