Get dashboard lineage groups

Returns lineage groups (sources, destinations, or transforms) with aggregated metrics. Use the group_by parameter to select the grouping dimension.

GET/dashboard/lineage

Authorization

AuthorizationBearer token · headerrequired

Query parameters

group_bystring

Group lineage by entity type

default: "source"

Allowed:sourcedestinationtransformsource_audit

searchstring

Search filter for group names

sortLineageGroupSortEnum

Sort field for lineage groups

sort_dirstring

Sort direction

default: "desc"

Allowed:ascdesc

pageinteger

Page number (1-indexed)

min 1 · default: 1

page_sizeinteger

Results per page

min 1 · max 100 · default: 10

include_metricsboolean

When false, skip ClickHouse metrics and return metrics as null for faster initial render

default: true

topic_namestring

Filter groups to those with matching topics (substring, case-insensitive)

source_idsstring

Comma-separated source entity IDs to filter groups by

destination_idsstring

Comma-separated destination entity IDs to filter groups by

transform_idsstring

Comma-separated transform entity IDs to filter groups by

include_topicsboolean

When true, embed first page of enriched topics in each group

default: false

topics_page_sizeinteger

Max topics to embed per group

min 1 · max 100 · default: 20

topics_sortstring

Sort field for embedded topics

default: "byteTotal"

Allowed:byteTotallatencyspeedlagname

topics_sort_dirstring

Sort direction for embedded topics

default: "desc"

Allowed:ascdesc

diff_bandsstring

Comma-separated diff% bands for topic filtering (audit tab). Each band is 'min-max' (inclusive). Multi-select: topic passes if its rounded-to-2-decimals max diff% falls in ANY band. Example: '0-0.05,2-10'.

latencystring

Filter groups by max_latency_ms using "op:value" syntax (e.g. "gt:300000"). Ops: gt|gte|lt|lte|eq|ne.

lagstring

Filter groups by total_lag using "op:value" syntax (e.g. "gt:1000"). Ops: gt|gte|lt|lte|eq|ne.

snapshotstring

Filter groups by snapshot state. Comma-separated list: running|pending|completed|failed|not_started.

Responses

200

Successful Response

pageinteger

Current page number

default: 1

page_sizeinteger

Results per page

default: 10

totalinteger | null

Total number of results

Show properties

Any of:

integer

integer

null

null

has_nextboolean | null

Whether more pages exist

Show properties

Any of:

boolean

boolean

null

null

resultLineageGroupRes[]

List of lineage groups

Show properties

Array of LineageGroupRes

idstringrequired

Group ID

entity_idstringrequired

Entity ID

namestringrequired

Display name

typestringrequired

Entity type: 'source', 'destination', or 'transform'

connectorstring

Connector type identifier

default: ""

statusstring

Current status

default: ""

topic_countinteger

Number of associated topics. Reflects filtered count when topic_name and include_topics are both set.

default: 0

total_topic_countinteger

Total number of topics regardless of topic_name filter

default: 0

metricsLineageGroupMetricsRes | null

Group-level metrics. Null when include_metrics=false.

Show properties

Any of:

LineageGroupMetricsRes

max_latency_msinteger

Maximum latency in milliseconds

default: 0

total_speed_rpsnumber

Total speed in records per second

default: 0

total_laginteger

Total consumer lag across all topics (sum)

default: 0

max_dest_latency_msinteger

Worst-partition destination wall-clock consumer latency (ms) across all reachable dest topics.

default: 0

topic_countinteger

Number of topics

default: 0

error_countinteger

Number of errors

default: 0

null

null

snapshot_progressSourceSnapshotProgressRes | null

Snapshot progress. Only present for source groups when include_metrics=true.

Show properties

Any of:

SourceSnapshotProgressRes

statestringrequired

Snapshot state: running, pending, completed, failed, or not_started

snapshot_typestring

Snapshot type: incremental or blocking

default: "incremental"

total_topic_countinteger

Logical topics (selectors) in the snapshot — what the user sees. Equals total_table_count unless a selector fanned out (sharded/regex).

default: 0

remaining_topic_countinteger

Logical topics still to be processed

default: 0

done_topic_countinteger

Logical topics in a terminal DONE/COMPLETED state

default: 0

cancelled_topic_countinteger

Logical topics cancelled by the user before completing

default: 0

failed_topic_countinteger

Logical topics that terminated in a FAILED state

default: 0

total_table_countinteger

Physical tables in the snapshot — expanded per shard/regex match. Exceeds total_topic_count only on fanout.

default: 0

remaining_table_countinteger

Physical tables still to be processed

default: 0

rows_totalinteger

Total source rows across all tables (from QA metrics). 0 if unavailable.

default: 0

rows_scannedinteger

Total rows scanned across all tables. 0 if unavailable.

default: 0

progress_typestring

Whether percent_complete is based on row counts ('rows') or table counts ('tables').

default: "tables"

Allowed:rowstables

percent_completenumber

Snapshot completion percentage (0-100)

default: 0

eta_msinteger

Estimated milliseconds to completion

default: 0

start_timestampstring | null

Snapshot start time (UTC)

Show properties

Any of:

string

string

null

null

end_timestampstring | null

Snapshot end time (UTC)

Show properties

Any of:

string

string

null

null

null

null

last_audit_atstring | null

ISO timestamp of the last completed QA metrics audit for this group. Only present for source_audit groups.

Show properties

Any of:

string

string

null

null

topicsEmbeddedTopicsRes | null

Embedded topics. Present when include_topics=true.

Show properties

Any of:

EmbeddedTopicsRes

resultLineageTopicRes[]

Enriched topics for this group

Show properties

Array of LineageTopicRes

idstringrequired

Topic entity ID

namestringrequired

Full topic name

sourceTopicSourceRefrequired

Reference to the source that produces a topic.

Show properties

idstringrequired

Source entity ID

namestringrequired

Source display name

connectorstring

Source connector type

default: ""

snapshot_progressSourceSnapshotProgressRes | null

Snapshot progress. Null when include_metrics=false or no snapshot data.

Show properties

Any of:

SourceSnapshotProgressRes

SourceSnapshotProgressRes

null

null

metricsTopicMetricsDetailRes | null

Topic-level metrics. Null when include_metrics=false.

Show properties

Any of:

TopicMetricsDetailRes

StreamingMilliSecondsBehindSourceinteger

Streaming latency in milliseconds

default: 0

SnapshotPercentageCompleteinteger

Snapshot completion percentage (uppercase)

default: 0

recordSendTotalinteger

Total records sent

default: 0

sourceRecordWriteTotalinteger

Total source records written

default: 0

snapshotPercentageCompleteinteger

Snapshot completion percentage (lowercase)

default: 0

recordsLaginteger

Consumer records lag

default: 0

byteTotalinteger

Total bytes processed

default: 0

sourceRecordPollRatenumber

Source record poll rate (records/sec)

default: 0

sourceQaCountnumber | null

Source DB row count from metrics_qa. Null when not collected.

Show properties

Any of:

number

number

null

null

null

null

snapshot_statusstring | null

Latest snapshot status for this topic: running, pending, completed, failed, or null if never snapshotted.

Show properties

Any of:

string

string

null

null

snapshot_detailsTopicSnapshotStatusRes[]

Detailed snapshot history for this topic. Empty list if never snapshotted.

Show properties

Array of TopicSnapshotStatusRes

statusstringrequired

Snapshot status: running, pending, completed, failed

start_timestampstring | null

Snapshot start time (UTC)

Show properties

Any of:

string

string

null

null

end_timestampstring | null

Snapshot end time (UTC)

Show properties

Any of:

string

string

null

null

duration_msinteger | null

Snapshot duration in milliseconds

Show properties

Any of:

integer

integer

null

null

rows_scannedinteger

Number of rows scanned so far

default: 0

rows_totalinteger

Total number of rows to scan

default: 0

tracestring | null

Error trace message for failed snapshots

Show properties

Any of:

string

string

null

null

eta_msinteger

Estimated milliseconds to completion

default: 0

transformsTopicTransformRef[]

Transforms applied to this topic

Show properties

Array of TopicTransformRef

idstringrequired

Transform entity ID

namestringrequired

Transform display name

statusstring

Transform status (e.g. RUNNING)

default: ""

metricsTransformMetricsRes | null

Transform-level metrics. Null when include_metrics=false.

Show properties

Any of:

TransformMetricsRes

latencyinteger

Transform processing latency in milliseconds

default: 0

null

null

output_topicsstring[]

Output topics from this transform

transform_typeTransformsLibraryEnum | null

Transform library type (e.g. topic_router, sql_join)

Show properties

Any of:

TransformsLibraryEnum

string

null

null

destinationsTopicDestinationRef[]

Destinations consuming this transform's output topics, with lag/count keyed to the output topic.

destinationsTopicDestinationRef[]

Destinations consuming this topic

Show properties

Array of TopicDestinationRef

idstringrequired

Destination entity ID

namestringrequired

Destination display name

connectorstring

Destination connector type

default: ""

pipeline_idstring

Pipeline entity ID

default: ""

pipeline_statusstring

Pipeline status

default: ""

laginteger | null

Destination consumer lag. Null when include_metrics=false.

Show properties

Any of:

integer

integer

null

null

latency_msinteger | null

Destination wall-clock consumer latency in milliseconds (worst partition). Null when include_metrics=false or no probe sample exists yet.

Show properties

Any of:

integer

integer

null

null

qaCountnumber | null

Destination row count for this topic from metrics_qa. Null when not collected.

Show properties

Any of:

number

number

null

null

e2e_eta_msinteger | null

Estimated milliseconds for destination to fully consume remaining data. Shown during active snapshots or when lag exceeds threshold. -1 means records are remaining but destination is not consuming (stalled). Null when not applicable.

Show properties

Any of:

integer

integer

null

null

totalinteger

Total matching topics in this group

default: 0

pageinteger

Page number (always 1 for embedded)

default: 1

has_nextboolean

Whether more topics exist beyond this page

default: false

null

null

422

Validation Error

detailValidationError[]

Show properties

Array of ValidationError

locstring | integer[]required

Show properties

Array of string | integer

Any of:

string

string

integer

integer

msgstringrequired

typestringrequired

inputany

ctxobject

Request

curl -X GET 'https://api.streamkap.com/dashboard/lineage' \
  -H 'Authorization: Bearer YOUR_TOKEN'

const response = await fetch("https://api.streamkap.com/dashboard/lineage", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN"
  }
});

import requests

response = requests.get(
    "https://api.streamkap.com/dashboard/lineage",
    headers={
        "Authorization": "Bearer YOUR_TOKEN"
    },
)

Response

{
  "page": 1,
  "page_size": 10,
  "total": 0,
  "has_next": true,
  "result": [
    {
      "id": "string",
      "entity_id": "string",
      "name": "string",
      "type": "string",
      "connector": "",
      "status": "",
      "topic_count": 0,
      "total_topic_count": 0,
      "metrics": {
        "max_latency_ms": 0,
        "total_speed_rps": 0,
        "total_lag": 0,
        "max_dest_latency_ms": 0,
        "topic_count": 0,
        "error_count": 0
      },
      "snapshot_progress": {
        "state": "string",
        "snapshot_type": "incremental",
        "total_topic_count": 0,
        "remaining_topic_count": 0,
        "done_topic_count": 0,
        "cancelled_topic_count": 0,
        "failed_topic_count": 0,
        "total_table_count": 0,
        "remaining_table_count": 0,
        "rows_total": 0,
        "rows_scanned": 0,
        "progress_type": "rows",
        "percent_complete": 0,
        "eta_ms": 0,
        "start_timestamp": "string",
        "end_timestamp": "string"
      },
      "last_audit_at": "string",
      "topics": {
        "result": [
          {
            "id": "string",
            "name": "string",
            "source": {
              "id": "string",
              "name": "string",
              "connector": "",
              "snapshot_progress": {
                "state": "string",
                "snapshot_type": "incremental",
                "total_topic_count": 0,
                "remaining_topic_count": 0,
                "done_topic_count": 0,
                "cancelled_topic_count": 0,
                "failed_topic_count": 0,
                "total_table_count": 0,
                "remaining_table_count": 0,
                "rows_total": 0,
                "rows_scanned": 0,
                "progress_type": "rows",
                "percent_complete": 0,
                "eta_ms": 0,
                "start_timestamp": "string",
                "end_timestamp": "string"
              }
            },
            "metrics": {
              "StreamingMilliSecondsBehindSource": 0,
              "SnapshotPercentageComplete": 0,
              "recordSendTotal": 0,
              "sourceRecordWriteTotal": 0,
              "snapshotPercentageComplete": 0,
              "recordsLag": 0,
              "byteTotal": 0,
              "sourceRecordPollRate": 0,
              "sourceQaCount": 0
            },
            "snapshot_status": "string",
            "snapshot_details": [
              {
                "status": "string",
                "start_timestamp": "string",
                "end_timestamp": "string",
                "duration_ms": 0,
                "rows_scanned": 0,
                "rows_total": 0,
                "trace": "string",
                "eta_ms": 0
              }
            ],
            "transforms": [
              {
                "id": "string",
                "name": "string",
                "status": "",
                "metrics": {
                  "latency": 0
                },
                "output_topics": [
                  "string"
                ],
                "transform_type": "fan_out",
                "destinations": [
                  {
                    "id": "string",
                    "name": "string",
                    "connector": "",
                    "pipeline_id": "",
                    "pipeline_status": "",
                    "lag": 0,
                    "latency_ms": 0,
                    "qaCount": 0,
                    "e2e_eta_ms": 0
                  }
                ]
              }
            ],
            "destinations": [
              {
                "id": "string",
                "name": "string",
                "connector": "",
                "pipeline_id": "",
                "pipeline_status": "",
                "lag": 0,
                "latency_ms": 0,
                "qaCount": 0,
                "e2e_eta_ms": 0
              }
            ]
          }
        ],
        "total": 0,
        "page": 1,
        "has_next": false
      }
    }
  ]
}

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string",
      "input": null,
      "ctx": {}
    }
  ]
}