Skip to content
Streamkap
Esc
navigateopen⌘Jpreview

Test Run Agent

Run an agent against a single sample record without deploying to Flink.

Synchronous, hard-capped at 15s wall-clock. Calls the LLM provider directly + dispatches MCP and HTTP tools live; transform tools and memory are intentionally not exercised (the Flink runtime owns them). Per-tenant rate limit: shares the validate-llm bucket (30 req/min/tenant) -> 429.

Provider failures (timeouts, schema errors, unsupported provider) land as a structured error object on the envelope (HTTP still 200) so the FE can render the partial timeline alongside the error context. Only the input-token cap raises (422) - at that point we can’t even start the run.

POST/agents/test-run
Authorization
AuthorizationBearer token · headerrequired
Request body
requiredapplication/json
configCreateAgentConfigRequestrequired

Request body for POST /agents/config - create a config-based agent.

Show properties
namestringrequired

Agent display name

min length 1 · max length 200
descriptionstring | any

Human-readable context for the agent

Show properties
Any of:
string
string
any
any
enabledboolean

Kill switch. When false, the Flink runtime accepts the deploy but stops emitting output — used by support to disable a runaway tenant agent without a full cancel/redeploy.

default: true
agentTypeAgentTypeEnum

Agent type — derived from the tool list post-validation; client-supplied value is overwritten

Allowed:workflowreact
inputAgentInputConfigrequired

Input configuration for the agent.

Show properties
topicPatternstringrequired

Regex pattern for input topics, e.g. '^(orders)$'

min length 1 · max length 1024
inputSerializationAgentSerializationEnum

Input deserialization format

Allowed:JsonConfluentAvroConfluent
fieldsstring[] | any

Column-level filter - only these fields sent to LLM

Show properties
Any of:
string[]
Array of string
string
any
any
createTableSQLstring | any

Flink SQL CREATE TABLE (required if filterSQL is set)

Show properties
Any of:
string
string
any
any
filterSQLstring | any

WHERE clause predicate for row-level filtering

Show properties
Any of:
string
string
any
any
outputAgentOutputConfigrequired

Output configuration for the agent.

Show properties
topicstringrequired

Fixed output topic name

min length 1 · max length 249
deadLetterTopicstring | any

DLQ topic name

Show properties
Any of:
string
string
any
any
outputSerializationAgentSerializationEnum

Output serialization format

Allowed:JsonConfluentAvroConfluent
schemaobject

Output schema: {field_name: TYPE_STRING}

llmAgentLlmConfigrequired

Inline LLM configuration embedded on the agent.

Carries provider, credentials, model, and tuning controls as a single blob. The FE prefills the form from a saved :class:AgentLlmConnection (which now carries default model + tuning + reasoning controls) and the user can override per-agent before save; the resulting blob is stored verbatim and shipped to the Flink runtime at deploy.

Two credential shapes (XOR, mirrors :class:ValidateLlmRequest):

  1. Inline - apiKey is filled directly. The BE KMS-encrypts it at save and decrypts it at deploy / test-run.
  2. Linked - llmConnectionId references a row in agent_connections.llmConnections and apiKey is left empty. The BE resolves the real key server-side at deploy and at test-run via :func:agents_service.resolve_saved_llm_credentials, so the browser never needs to hold (or re-paste) the stored plaintext when editing an existing agent.

provider == ollama accepts an empty key without a link (the runtime is local and unauthenticated).

Show properties
providerAgentLlmProviderEnumrequired

Unified LLM provider enum.

A single AgentLlmConnection row carries one provider and a set of capabilities (chat / embedding). PROVIDER_CAPABILITIES below pins which capabilities each provider can serve — picked by the FE Connections drawer and re-validated server-side on every write.

Allowed:anthropicopenaiopenai-responsesollamaazureazure-openaibedrockqwenopenai-compatible
modelstringrequired

Model name, e.g. claude-sonnet-4-20250514

min length 1
apiKeystring

API key - use '${SECRET:ENV_VAR_NAME}' for env var resolution. Empty when llmConnectionId is set.

max length 512 · default: ""
llmConnectionIdstring | any

Tenant-scoped reference to agent_connections.llmConnections[].id. When set with an empty apiKey, the BE resolves the real key server-side at deploy and at test-run.

Show properties
Any of:
string
string
any
any
baseUrlstring | any

Base URL for OpenAI-compatible proxies

Show properties
Any of:
string
string
any
any
temperaturenumber | any

Sampling temperature (null = matrix-skipped)

Show properties
Any of:
number
number
any
any
maxTokensinteger | any

Max output tokens (null = matrix-skipped)

Show properties
Any of:
integer
integer
any
any
timeoutinteger | any

Request timeout in seconds (null = matrix-skipped)

Show properties
Any of:
integer
integer
any
any
reasoningEffortstring | any

OpenAI / GPT-5 reasoning effort. The accepted set is per-model: the matrix in app/utils/llm_capabilities.py rejects values not in caps.reasoning_effort_values for the chosen provider+model.

Show properties
Any of:
string
string
any
any
thinkingBudgetTokensinteger | any

Anthropic Claude 4 extended-thinking budget (tokens). Non-null enables thinking.

Show properties
Any of:
integer
integer
any
any
ollamaThinkboolean | any

Ollama 'think' toggle for reasoning models

Show properties
Any of:
boolean
boolean
any
any
maxRetriesinteger | any

Universal — max retry attempts on LLM call. Accepted by every provider.

Show properties
Any of:
integer
integer
any
any
regionstring | any

Bedrock: AWS region (e.g. us-east-1). Ignored for non-bedrock providers.

Show properties
Any of:
string
string
any
any
strictboolean | any

openai-responses: enable JSON-schema strict mode.

Show properties
Any of:
boolean
boolean
any
any
storeboolean | any

openai-responses: server-side response storage flag.

Show properties
Any of:
boolean
boolean
any
any
instructionsstring | any

openai-responses: system-level instructions passed as a top-level Responses param.

Show properties
Any of:
string
string
any
any
additionalKwargsobject | any

openai-responses / azure-openai: free-form extra request params forwarded verbatim by the runtime.

Show properties
Any of:
object
object
any
any
apiVersionstring | any

azure-openai: Azure OpenAI API version (e.g. '2024-02-01'). Required for azure-openai.

Show properties
Any of:
string
string
any
any
azureEndpointstring | any

azure-openai: Azure resource endpoint (e.g. https://<resource>.openai.azure.com). Required.

Show properties
Any of:
string
string
any
any
azureUrlPathModestring | any

azure-openai: URL path resolution — AUTO / LEGACY / UNIFIED. Optional.

Show properties
Any of:
string
string
any
any
promptsAgentPromptConfigrequired

Prompt configuration for the agent.

The final system prompt is auto-assembled by build_system_prompt() from the agent type template + output schema + tool descriptions + custom_instructions. The system field stores the assembled result (set by the backend, not the user).

Show properties
systemstring

Assembled system prompt (auto-generated by backend)

max length 8000 · default: ""
customInstructionsstring

User-provided instructions appended to the auto-generated prompt

max length 8000 · default: ""
userstring

User prompt template - {input_json} is replaced with the input record

max length 2000 · default: "<record>{input_json}</record>"
mcpServerAgentMcpServerConfig | any

Shared MCP server config for all MCP tools

Show properties
Any of:
AgentMcpServerConfig
projectKeyIdstring | any

ID of an agentic-enabled Project Key. When set, deploy-time resolves the Streamkap MCP serverUrl + auth header from the PK row. Mutually exclusive with serverUrl - one or the other, never both.

Show properties
Any of:
string
string
any
any
serverUrlstring | any

External MCP server URL. Required when projectKeyId is unset. Server-stamped from the PK row when projectKeyId is set.

Show properties
Any of:
string
string
any
any
headersobject

Auth headers for MCP server

any
any
toolsAgentToolConfig[]

Agent tools (MCP, HTTP, transform)

Show properties
Array of AgentToolConfig
namestringrequired

Tool name (must be unique)

min length 1 · max length 100
typeAgentToolTypeEnumrequired

Tool type: http, transform, or mcp

Allowed:httptransformmcp
descriptionstring

Tool description for the LLM

max length 2000 · default: ""
configobject

Type-specific configuration

parametersAgentToolParameterConfig[]

Tool parameters the LLM can provide

Show properties
Array of AgentToolParameterConfig
namestringrequired

Parameter name

min length 1 · max length 100
typestring

Parameter type (string, number, boolean)

max length 50 · default: "string"
descriptionstring

Parameter description for the LLM

max length 2000 · default: ""
memoryAgentMemoryConfig | any

Memory config (short-term + long-term)

Show properties
Any of:
AgentMemoryConfig
keyFieldstring | any

Field to partition memory by (e.g. 'customer_id'). Default: topic key

Show properties
Any of:
string
string
any
any
shortTermAgentShortTermMemoryConfig

Short-term (conversation) memory config.

Show properties
enabledboolean

Enable short-term memory

default: false
ttlMsinteger

Time-to-live in milliseconds (default: 1 hour)

min 0 · default: 3600000
maxEntriesinteger

Max conversation entries per key

min 1 · max 100 · default: 20
longTermAgentLongTermMemoryConfig

Long-term (vector store) memory config.

References a saved AgentVectorStoreConnection by id. At deploy time the resolver reads the connection's apiKey + endpoint and composes the inline vectorStore block the Java runtime expects.

Legacy agents may still carry destinationId (a Pinecone destination connector from the old flow). The deploy resolver handles both: if vectorStoreConnectionId is set it wins; otherwise it falls back to the legacy destinationId path. New agents created through the FE always use vectorStoreConnectionId.

Show properties
enabledboolean

Enable long-term memory

default: false
vectorStoreConnectionIdstring | any

Reference to a saved AgentVectorStoreConnection. Resolved at deploy time.

Show properties
Any of:
string
string
any
any
namespacestring | any

Per-agent namespace override (falls back to connection's defaultNamespace)

Show properties
Any of:
string
string
any
any
destinationIdstring | any

(Legacy) ObjectId of a Pinecone destination connector. Prefer vectorStoreConnectionId.

Show properties
Any of:
string
string
any
any
any
any
knowledgeBasesAgentKnowledgeBaseRef[]

Knowledge bases this agent can query at runtime for RAG

max items 10
Show properties
Array of AgentKnowledgeBaseRef
idstringrequired

Knowledge base entity ID

min length 1 · max length 64
namestring

Display name (denormalized for UI)

max length 200 · default: ""
processingAgentProcessingConfig

Processing configuration for the agent.

Show properties
parallelisminteger

Parallelism

min 1 · max 128 · default: 1
checkpointIntervalMininteger

Checkpoint interval in minutes

min 1 · max 60 · default: 5
maxIterationsinteger

Max tool-call iterations per record (react type only)

min 1 · max 25 · default: 10
maxTokensPerHourinteger

Per-TaskManager rolling-hour token budget. 0 = unlimited. When exceeded, records pass through with a _budget_exceeded: true marker.

min 0 · default: 0
sampleRecordobjectrequired

One Kafka-shaped record to feed the agent

agentIdstring | any

Resolves an empty / masked apiKey against the stored agent's encrypted blob. Post-cutover the wizard sends apiKey: '' for every existing-agent test-run; this lets the BE swap in the stored key for legacy inline-key agents that were saved before the Connection cutover (no llmConnectionId to resolve via).

Show properties
Any of:
string
string
any
any
savedConnectionIdstring | any

Mirror of the ValidateLlmRequest.savedConnectionId path.

Show properties
Any of:
string
string
any
any
skipToolsboolean

When true, every MCP / HTTP tool dispatch returns a canned {skipped: true} payload to the LLM. Use to iterate on prompt shape without burning real tool calls.

default: false
Responses
200

Successful Response

stepsTestRunStep[]required
Show properties
Array of TestRunStep
kindstringrequired
Allowed:llmtool
modelstring | any
Show properties
Any of:
string
string
any
any
tokensIninteger | any
Show properties
Any of:
integer
integer
any
any
tokensOutinteger | any
Show properties
Any of:
integer
integer
any
any
stopReasonstring | any
Show properties
Any of:
string
string
any
any
textstring | any
Show properties
Any of:
string
string
any
any
toolCallsobject[] | any
Show properties
Any of:
object[]
Array of object
object
any
any
toolNamestring | any
Show properties
Any of:
string
string
any
any
toolTypestring | any
Show properties
Any of:
string
string
any
any
requestobject | any
Show properties
Any of:
object
object
any
any
resultobject | any
Show properties
Any of:
object
object
any
any
skippedboolean | any
Show properties
Any of:
boolean
boolean
any
any
skippedReasonstring | any
Show properties
Any of:
string
string
any
any
durationMsinteger | any
Show properties
Any of:
integer
integer
any
any
errorstring | any
Show properties
Any of:
string
string
any
any
finalOutputany
stoppedReasonstringrequired
Allowed:completedmax_iterationserrortimeout
iterationsUsedintegerrequired
tokensInintegerrequired
tokensOutintegerrequired
durationMsintegerrequired
approximationsstring[]

Free-form list of caveats, e.g. 'memory_excluded', 'transforms_skipped'.

suggestionsTestRunSuggestion[]

Deterministic hints (max 4) computed from the agent config + run result. Empty when the run produced a useful structured output. Each suggestion has a stable kind the FE can map to icon / copy.

Show properties
Array of TestRunSuggestion
kindstringrequired
Allowed:missing_output_schematools_not_invokednear_empty_responseskip_instruction_may_applymax_tokens_truncated
messagestringrequired
warningsTestRunWarning[]

Static config-vs-record checks (e.g. memory keyField missing on the sample) that the Flink runtime would silently swallow at deploy. Empty when nothing is amiss.

Show properties
Array of TestRunWarning
kindstringrequired
Allowed:memory_key_field_missinginput_fields_not_in_recordmax_tokens_clamped
messagestringrequired
outputSchemaCheckTestRunSchemaCheck | any

Field-level diff between the declared output.schema and the LLM's finalOutput. Null when no schema was declared OR the run errored / produced a non-structured (text envelope) result. ok=true means every declared field is present with a compatible type.

Show properties
Any of:
TestRunSchemaCheck
okbooleanrequired
missingstring[]

Fields declared in schema but absent from finalOutput

extrastring[]

Fields present in finalOutput but not declared

typeMismatchesTestRunSchemaTypeMismatch[]

Fields whose value types diverge from the declared schema

Show properties
Array of TestRunSchemaTypeMismatch
fieldstringrequired
expectedstringrequired

Declared schema type, e.g. STRING / DOUBLE

actualstringrequired

Inferred type of the value the LLM returned

any
any
systemPromptstring | any

Composed system prompt the LLM actually received - base template + auto-injected schema fields + tool descriptions + customInstructions. Identical to what Flink sends at deploy time (built by the same build_system_prompt). Null when the run errored before reaching the LLM (e.g. llm_resolution_error).

Show properties
Any of:
string
string
any
any
userMessagestring | any

First user message dispatched to the LLM, after the prompts.user template wraps the (input-fields-filtered) sample record. Lets the test pane show the exact pair of strings the LLM saw on iteration 1. Null when the run errored before message construction.

Show properties
Any of:
string
string
any
any
errorTestRunError | any
Show properties
Any of:
TestRunError
kindstringrequired

One of: llm_resolution_error, llm_provider_error, timeout, unsupported_provider, http_connection_resolution_error, …

messagestringrequired
any
any
422

Validation Error

detailValidationError[]
Show properties
Array of ValidationError
locstring | integer[]required
Show properties
Array of string | integer
Any of:
string
string
integer
integer
msgstringrequired
typestringrequired
inputany
ctxobject
Request
curl -X POST "https://api.streamkap.com/agents/test-run" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "config": {
    "name": "string",
    "description": "string",
    "enabled": true,
    "agentType": "workflow",
    "input": {
      "topicPattern": "string",
      "inputSerialization": "JsonConfluent",
      "fields": [
        "string"
      ],
      "createTableSQL": "string",
      "filterSQL": "string"
    },
    "output": {
      "topic": "string",
      "deadLetterTopic": "string",
      "outputSerialization": "JsonConfluent",
      "schema": {
        "property1": "STRING",
        "property2": "STRING"
      }
    },
    "llm": {
      "provider": "anthropic",
      "model": "string",
      "apiKey": "",
      "llmConnectionId": "string",
      "baseUrl": "string",
      "temperature": 0,
      "maxTokens": 1,
      "timeout": 1,
      "reasoningEffort": "none",
      "thinkingBudgetTokens": 0,
      "ollamaThink": true,
      "maxRetries": 0,
      "region": "string",
      "strict": true,
      "store": true,
      "instructions": "string",
      "additionalKwargs": {},
      "apiVersion": "string",
      "azureEndpoint": "string",
      "azureUrlPathMode": "AUTO"
    },
    "prompts": {
      "system": "",
      "customInstructions": "",
      "user": "<record>{input_json}</record>"
    },
    "mcpServer": {
      "projectKeyId": "string",
      "serverUrl": "string",
      "headers": {
        "property1": "string",
        "property2": "string"
      }
    },
    "tools": [
      {
        "name": "string",
        "type": "http",
        "description": "",
        "config": {},
        "parameters": [
          {
            "name": "string",
            "type": "string",
            "description": ""
          }
        ]
      }
    ],
    "memory": {
      "keyField": "string",
      "shortTerm": {
        "enabled": false,
        "ttlMs": 3600000,
        "maxEntries": 20
      },
      "longTerm": {
        "enabled": false,
        "vectorStoreConnectionId": "string",
        "namespace": "string",
        "destinationId": "string"
      }
    },
    "knowledgeBases": [
      {
        "id": "string",
        "name": ""
      }
    ],
    "processing": {
      "parallelism": 1,
      "checkpointIntervalMin": 5,
      "maxIterations": 10,
      "maxTokensPerHour": 0
    }
  },
  "sampleRecord": {},
  "agentId": "string",
  "savedConnectionId": "string",
  "skipTools": false
}'
Response
{
  "steps": [
    {
      "kind": "llm",
      "model": "string",
      "tokensIn": 0,
      "tokensOut": 0,
      "stopReason": "string",
      "text": "string",
      "toolCalls": [
        {}
      ],
      "toolName": "string",
      "toolType": "string",
      "request": {},
      "result": {},
      "skipped": true,
      "skippedReason": "string",
      "durationMs": 0,
      "error": "string"
    }
  ],
  "finalOutput": null,
  "stoppedReason": "completed",
  "iterationsUsed": 0,
  "tokensIn": 0,
  "tokensOut": 0,
  "durationMs": 0,
  "approximations": [
    "string"
  ],
  "suggestions": [
    {
      "kind": "missing_output_schema",
      "message": "string"
    }
  ],
  "warnings": [
    {
      "kind": "memory_key_field_missing",
      "message": "string"
    }
  ],
  "outputSchemaCheck": {
    "ok": true,
    "missing": [
      "string"
    ],
    "extra": [
      "string"
    ],
    "typeMismatches": [
      {
        "field": "string",
        "expected": "string",
        "actual": "string"
      }
    ]
  },
  "systemPrompt": "string",
  "userMessage": "string",
  "error": {
    "kind": "string",
    "message": "string"
  }
}