Get dashboard lineage groups

Returns lineage groups (sources, destinations, or transforms) with aggregated metrics. Use the group_by parameter to select the grouping dimension.

GET/dashboard/lineage

Authorization

AuthorizationBearer token · headerrequired

Query parameters

group_bystring

Group lineage by entity type

default: "source"

Allowed:sourcedestinationtransformsource_audit

searchstring

Search filter for group names

sortLineageGroupSortEnum

Sort field for lineage groups

sort_dirstring

Sort direction

default: "desc"

Allowed:ascdesc

pageinteger

Page number (1-indexed)

min 1 · default: 1

page_sizeinteger

Results per page

min 1 · max 100 · default: 10

include_metricsboolean

When false, skip ClickHouse metrics and return metrics as null for faster initial render

default: true

topic_namestring

Filter groups to those with matching topics (substring, case-insensitive)

source_idsstring

Comma-separated source entity IDs to filter groups by

destination_idsstring

Comma-separated destination entity IDs to filter groups by

transform_idsstring

Comma-separated transform entity IDs to filter groups by

include_topicsboolean

When true, embed first page of enriched topics in each group

default: false

topics_page_sizeinteger

Max topics to embed per group

min 1 · max 100 · default: 20

topics_sortstring

Sort field for embedded topics

default: "byteTotal"

Allowed:byteTotallatencyspeedlagname

topics_sort_dirstring

Sort direction for embedded topics

default: "desc"

Allowed:ascdesc

diff_bandsstring

Comma-separated diff% bands for topic filtering (audit tab). Each band is 'min-max' (inclusive). Multi-select: topic passes if its rounded-to-2-decimals max diff% falls in ANY band. Example: '0-0.05,2-10'.

latencystring

Filter groups by max_latency_ms using "op:value" syntax (e.g. "gt:300000"). Ops: gt|gte|lt|lte|eq|ne.

lagstring

Filter groups by total_lag using "op:value" syntax (e.g. "gt:1000"). Ops: gt|gte|lt|lte|eq|ne.

snapshotstring

Responses

200

Successful Response

pageinteger

Current page number

default: 1

page_sizeinteger

Results per page

default: 10

totalinteger | null

Total number of results

Show properties

Any of:

integer

integer

null

null

has_nextboolean | null

Whether more pages exist

Show properties

Any of:

boolean

boolean

null

null

resultLineageGroupRes[]

List of lineage groups

Show properties

Array of LineageGroupRes

idstringrequired

Group ID

entity_idstringrequired

Entity ID

namestringrequired

Display name

typestringrequired

Entity type: 'source', 'destination', or 'transform'

connectorstring

Connector type identifier

default: ""

statusstring

Current status

default: ""

topic_countinteger

Number of associated topics. Reflects filtered count when topic_name and include_topics are both set.

default: 0

total_topic_countinteger

Total number of topics regardless of topic_name filter

default: 0

metricsLineageGroupMetricsRes | null

Group-level metrics. Null when include_metrics=false.

Show properties

Any of:

LineageGroupMetricsRes

max_latency_msinteger

Maximum latency in milliseconds

default: 0

total_speed_rpsnumber

Total speed in records per second

default: 0

total_laginteger

Total consumer lag across all topics (sum)

default: 0

max_dest_latency_msinteger

Worst-partition destination wall-clock consumer latency (ms) across all reachable dest topics.

default: 0

topic_countinteger

Number of topics

default: 0

error_countinteger

Number of errors

default: 0

null

null

snapshot_progressSourceSnapshotProgressRes | null

Snapshot progress. Only present for source groups when include_metrics=true.

Show properties

Any of:

SourceSnapshotProgressRes

statestringrequired

Snapshot state: running, pending, completed, failed, or not_started

snapshot_typestring

Snapshot type: incremental or blocking

default: "incremental"

total_topic_countinteger

Logical topics (selectors) in the snapshot — what the user sees. Equals total_table_count unless a selector fanned out (sharded/regex).

default: 0

remaining_topic_countinteger

Logical topics still to be processed

default: 0

done_topic_countinteger

Logical topics in a terminal DONE/COMPLETED state

default: 0

cancelled_topic_countinteger

Logical topics cancelled by the user before completing

default: 0

failed_topic_countinteger

Logical topics that terminated in a FAILED state

default: 0

total_table_countinteger

Physical tables in the snapshot — expanded per shard/regex match. Exceeds total_topic_count only on fanout.

default: 0

remaining_table_countinteger

Physical tables still to be processed

default: 0

rows_totalinteger

Total source rows across all tables (from QA metrics). 0 if unavailable.

default: 0

rows_scannedinteger

Total rows scanned across all tables. 0 if unavailable.

default: 0

progress_typestring

Whether percent_complete is based on row counts ('rows') or table counts ('tables').

default: "tables"

Allowed:rowstables

percent_completenumber

Snapshot completion percentage (0-100)

default: 0

eta_msinteger

Estimated milliseconds to completion

default: 0

start_timestampstring | null

Snapshot start time (UTC)

Show properties

Any of:

string

string

null

null

end_timestampstring | null

Snapshot end time (UTC)

Show properties

Any of:

string

string

null

null

null

null

last_audit_atstring | null

ISO timestamp of the last completed QA metrics audit for this group. Only present for source_audit groups.

Show properties

Any of:

string

string

null

null

topicsEmbeddedTopicsRes | null

Embedded topics. Present when include_topics=true.

Show properties

Any of:

EmbeddedTopicsRes

resultLineageTopicRes[]

Enriched topics for this group

Show properties

Array of LineageTopicRes

idstringrequired

Topic entity ID

namestringrequired

Full topic name

sourceTopicSourceRefrequired

Source that produces this topic

Show properties

idstringrequired

Source entity ID

namestringrequired

Source display name

connectorstring

Source connector type

default: ""

snapshot_progressSourceSnapshotProgressRes | null

Snapshot progress. Null when include_metrics=false or no snapshot data.

Show properties

Any of:

SourceSnapshotProgressRes

SourceSnapshotProgressRes

null

null

metricsTopicMetricsDetailRes | null

Topic-level metrics. Null when include_metrics=false.

Show properties

Any of:

TopicMetricsDetailRes

StreamingMilliSecondsBehindSourceinteger

Streaming latency in milliseconds

default: 0

SnapshotPercentageCompleteinteger

Snapshot completion percentage (uppercase)

default: 0

recordSendTotalinteger

Total records sent

default: 0

sourceRecordWriteTotalinteger

Total source records written

default: 0

snapshotPercentageCompleteinteger

Snapshot completion percentage (lowercase)

default: 0

recordsLaginteger

Consumer records lag

default: 0

byteTotalinteger

Total bytes processed

default: 0

sourceRecordPollRatenumber

Source record poll rate (records/sec)

default: 0

sourceQaCountnumber | null

Source DB row count from metrics_qa. Null when not collected.

Show properties

Any of:

number

number

null

null

null

null

snapshot_statusstring | null

Latest snapshot status for this topic: running, pending, completed, failed, or null if never snapshotted.

Show properties

Any of:

string

string

null

null

snapshot_detailsTopicSnapshotStatusRes[]

Detailed snapshot history for this topic. Empty list if never snapshotted.

Show properties

Array of TopicSnapshotStatusRes

statusstringrequired

Snapshot status: running, pending, completed, failed

start_timestampstring | null

Snapshot start time (UTC)

Show properties

Any of:

string

string

null

null

end_timestampstring | null

Snapshot end time (UTC)

Show properties

Any of:

string

string

null

null

duration_msinteger | null

Snapshot duration in milliseconds

Show properties

Any of:

integer

integer

null

null

rows_scannedinteger

Number of rows scanned so far

default: 0

rows_totalinteger

Total number of rows to scan

default: 0

tracestring | null

Error trace message for failed snapshots

Show properties

Any of:

string

string

null

null

eta_msinteger

Estimated milliseconds to completion

default: 0

transformsTopicTransformRef[]

Transforms applied to this topic

Show properties

Array of TopicTransformRef

idstringrequired

Transform entity ID

namestringrequired

Transform display name

statusstring

Transform status (e.g. RUNNING)

default: ""

metricsTransformMetricsRes | null

Transform-level metrics. Null when include_metrics=false.

Show properties

Any of:

TransformMetricsRes

latencyinteger

Transform processing latency in milliseconds

default: 0

null

null

output_topicsstring[]

Output topics from this transform

transform_typeTransformsLibraryEnum | null

Transform library type (e.g. topic_router, sql_join)

Show properties

Any of:

TransformsLibraryEnum

string

null

null

destinationsTopicDestinationRef[]

Destinations consuming this transform's output topics, with lag/count keyed to the output topic.

destinationsTopicDestinationRef[]

Destinations consuming this topic

Show properties

Array of TopicDestinationRef

idstringrequired

Destination entity ID

namestringrequired

Destination display name

connectorstring

Destination connector type

default: ""

pipeline_idstring

Pipeline entity ID

default: ""

pipeline_statusstring

Pipeline status

default: ""

laginteger | null

Destination consumer lag. Null when include_metrics=false.

Show properties

Any of:

integer

integer

null

null

latency_msinteger | null

Destination wall-clock consumer latency in milliseconds (worst partition). Null when include_metrics=false or no probe sample exists yet.

Show properties

Any of:

integer

integer

null

null

qaCountnumber | null

Destination row count for this topic from metrics_qa. Null when not collected.

Show properties

Any of:

number

number

null

null

e2e_eta_msinteger | null

Estimated milliseconds for destination to fully consume remaining data. Shown during active snapshots or when lag exceeds threshold. -1 means records are remaining but destination is not consuming (stalled). Null when not applicable.

Show properties

Any of:

integer

integer

null

null

totalinteger

Total matching topics in this group

default: 0

pageinteger

Page number (always 1 for embedded)

default: 1

has_nextboolean

Whether more topics exist beyond this page

default: false

null

null

422

Validation Error

detailValidationError[]

Show properties

Array of ValidationError

locstring | integer[]required

Show properties

Array of string | integer

Any of:

string

string

integer

integer

msgstringrequired

typestringrequired

inputany

ctxobject

Request

curl -X GET 'https://api.streamkap.com/dashboard/lineage' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Accept: application/json'

const response = await fetch("https://api.streamkap.com/dashboard/lineage", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json"
  }
});

import requests

response = requests.get(
    "https://api.streamkap.com/dashboard/lineage",
    headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Accept": "application/json"
    },
)

Response

{
  "page": 1,
  "page_size": 10,
  "total": 0,
  "has_next": true,
  "result": [
    {
      "id": "string",
      "entity_id": "string",
      "name": "string",
      "type": "string",
      "connector": "",
      "status": "",
      "topic_count": 0,
      "total_topic_count": 0,
      "metrics": {
        "max_latency_ms": 0,
        "total_speed_rps": 0,
        "total_lag": 0,
        "max_dest_latency_ms": 0,
        "topic_count": 0,
        "error_count": 0
      },
      "snapshot_progress": {
        "state": "string",
        "snapshot_type": "incremental",
        "total_topic_count": 0,
        "remaining_topic_count": 0,
        "done_topic_count": 0,
        "cancelled_topic_count": 0,
        "failed_topic_count": 0,
        "total_table_count": 0,
        "remaining_table_count": 0,
        "rows_total": 0,
        "rows_scanned": 0,
        "progress_type": "rows",
        "percent_complete": 0,
        "eta_ms": 0,
        "start_timestamp": "string",
        "end_timestamp": "string"
      },
      "last_audit_at": "string",
      "topics": {
        "result": [
          {
            "id": "string",
            "name": "string",
            "source": {
              "id": "string",
              "name": "string",
              "connector": "",
              "snapshot_progress": {
                "state": "string",
                "snapshot_type": "incremental",
                "total_topic_count": 0,
                "remaining_topic_count": 0,
                "done_topic_count": 0,
                "cancelled_topic_count": 0,
                "failed_topic_count": 0,
                "total_table_count": 0,
                "remaining_table_count": 0,
                "rows_total": 0,
                "rows_scanned": 0,
                "progress_type": "rows",
                "percent_complete": 0,
                "eta_ms": 0,
                "start_timestamp": "string",
                "end_timestamp": "string"
              }
            },
            "metrics": {
              "StreamingMilliSecondsBehindSource": 0,
              "SnapshotPercentageComplete": 0,
              "recordSendTotal": 0,
              "sourceRecordWriteTotal": 0,
              "snapshotPercentageComplete": 0,
              "recordsLag": 0,
              "byteTotal": 0,
              "sourceRecordPollRate": 0,
              "sourceQaCount": 0
            },
            "snapshot_status": "string",
            "snapshot_details": [
              {
                "status": "string",
                "start_timestamp": "string",
                "end_timestamp": "string",
                "duration_ms": 0,
                "rows_scanned": 0,
                "rows_total": 0,
                "trace": "string",
                "eta_ms": 0
              }
            ],
            "transforms": [
              {
                "id": "string",
                "name": "string",
                "status": "",
                "metrics": {
                  "latency": 0
                },
                "output_topics": [
                  "string"
                ],
                "transform_type": "fan_out",
                "destinations": [
                  {
                    "id": "string",
                    "name": "string",
                    "connector": "",
                    "pipeline_id": "",
                    "pipeline_status": "",
                    "lag": 0,
                    "latency_ms": 0,
                    "qaCount": 0,
                    "e2e_eta_ms": 0
                  }
                ]
              }
            ],
            "destinations": [
              {
                "id": "string",
                "name": "string",
                "connector": "",
                "pipeline_id": "",
                "pipeline_status": "",
                "lag": 0,
                "latency_ms": 0,
                "qaCount": 0,
                "e2e_eta_ms": 0
              }
            ]
          }
        ],
        "total": 0,
        "page": 1,
        "has_next": false
      }
    }
  ]
}

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string",
      "input": null,
      "ctx": {}
    }
  ]
}