Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Knowledge Bases

Stream a Kafka topic into a vector store as embeddings so Streamkap agents can retrieve from live data at run time

A knowledge base streams a Kafka topic through an embedding model into a vector store, keeping an always-current index your agents can retrieve from. Attach a knowledge base to an agent and it becomes a retrieval tool: the model decides when to look something up, and gets back the most relevant chunks from your data.

Knowledge bases live on the Knowledge Bases tab of the Agentic section. Creating and managing them requires the write:agents permission.

Prerequisites

  • An LLM connection with the Embedding capability — OpenAI, Azure, Ollama, AWS Bedrock, Qwen, or any OpenAI-compatible endpoint. See Connections.
  • A vector store connection — Pinecone, pgvector, Milvus, Amazon OpenSearch, Elasticsearch, or Amazon S3 Vectors. See Connections.
  • The input Kafka topic to embed.

Create a Knowledge Base

Creation is a single form (not a multi-step wizard). Open Knowledge Bases and click Create Knowledge Base.

Name and source

Enter a Name (auto-derived from the topic) and select the input topic and its serialization (auto-detected, JSON, or Avro). Optionally set a text template with ${field} placeholders to control what text gets embedded.

Embedding and destination

Pick the embedding model connection and the vector store connection, then set the index name (auto-derived) — the index or collection the embeddings are written to — and an optional namespace.

Chunking and runtime (advanced)

Optionally adjust chunk size and overlap, parallelism, checkpoint interval, and the distance metric (Cosine, Dot product, or Euclidean).

Create

Click Create Knowledge Base to deploy, or Save as draft to finish later.

Attach to an Agent

On the agent wizard’s Input step, use the Knowledge Bases picker to attach up to 10 running knowledge bases. The agent gets one retrieval tool per attached knowledge base, and the model retrieves from it when the prompt calls for it. See Build an Agent.

Preview Retrieval

From a knowledge base detail page, use Preview retrieval to test what the agent would get back. Enter a query and a result count (n), and Streamkap returns that many ranked chunks with their similarity scores and metadata.

Preview retrieval requires the knowledge base to be Running and the write:agents permission (retrieved chunks are your data). It’s currently supported for Pinecone vector stores (including the Streamkap default); knowledge bases on other vector stores still serve retrieval to agents at run time — only the in-app preview is unavailable.

Lifecycle

Knowledge bases reuse the same streaming infrastructure as agents but have a reduced lifecycle: Cancel, Stop, and Delete only. Because settings are immutable, there is no edit or redeploy. Recreate to change configuration.