Overview
The Pipelines page provides a centralized interface to:- View all pipelines with their status, performance metrics, and configurations
- Create new pipelines connecting sources to destinations
- Monitor pipeline health, latency, and lag
- Filter pipelines by connector, status, or tags
- Configure pipeline settings including schema selection and transforms
- Access source and destination connector details
Table Columns
The Pipelines table displays the following information:- Name: The pipeline name with checkbox for bulk selection
- Tags: Custom tags for organizing pipelines (e.g., Production, Development)
- Status: Current pipeline state with visual indicators:
- Active (green): Pipeline is running normally
- Stopped (gray): Pipeline is paused
- Broken (red with info icon): Pipeline has errors
- Latency: Time delay for data to flow through the pipeline
- Lag: Number of records behind the source
- Topics: Number of Kafka topics associated with the pipeline
- Source: Source connector type and name (clickable to view details)
- Destination: Destination connector type and name (clickable to view details)
- Actions: Three-dot menu button for quick actions
Pipeline Statuses
Pipelines can be in one of the following states:Active
The pipeline is running and actively replicating data from source to destination. Indicated by green status badge.Stopped
The pipeline has been manually paused or stopped. No data is flowing. Indicated by gray status badge.Broken
The pipeline has encountered an error and is not functioning. Click the info icon next to the status to see error details. Indicated by red status badge with an information icon.When a pipeline shows a Broken status, check the Alerts and Logs pages for detailed error information to troubleshoot the issue.
Filtering and Search
Use the filtering options at the top of the page to quickly find specific pipelines:Search
Type in the search box to filter pipelines by name. The search filters in real-time as you type.Filter by Connectors
Click the Connectors dropdown to filter pipelines by:- Source connector type (MySQL, PostgreSQL, MongoDB, etc.)
- Destination connector type (Snowflake, BigQuery, Databricks, etc.)
Filter by Status
Click the Status dropdown to show only pipelines with specific statuses:- Active
- Stopped
- Broken
Filter by Tags
Click the Tags dropdown to filter pipelines by assigned tags (Production, Development, custom tags).Quick Actions
Each pipeline row includes an action menu (three-dot button) on the right side with the following options:- View Details: Open the pipeline detail page
- Start/Stop: Start or stop the pipeline
- Delete: Remove the pipeline (requires confirmation)
Deleting a pipeline is permanent and will stop all data replication. The underlying source and destination connectors will remain unchanged.
Creating a Pipeline
To create a new pipeline:1
Click Create
Click the Create button at the top left of the Pipelines page.
2
Select Source
Choose an existing source connector or create a new one.Source connectors include:
- MySQL
- PostgreSQL
- MongoDB
- SQL Server
- DynamoDB
- And others
3
Select Destination
Choose an existing destination connector or create a new one.Destination connectors include:
- Snowflake
- BigQuery
- Databricks
- Iceberg
- ClickHouse
- And others
4
Configure Pipeline
Set the pipeline name, tags, and other configuration options in the Settings tab.
5
Select Schema
Choose which databases and tables from the source to replicate.
6
Apply Transforms (Optional)
Add transforms to modify data as it flows through the pipeline.
7
Save and Start
Save the pipeline configuration and start data replication.
Pipeline Detail View
Click on any pipeline name to view its detail page with comprehensive information and configuration options.Status Tab
The Status tab provides an overview of the pipeline’s current state and performance:
- Displays all tags assigned to the pipeline
- Tags help organize and categorize pipelines
- Status: Current pipeline state (Active, Stopped, or Broken)
- Topics: Number of Kafka topics in the pipeline
- Latency: Current latency measurement
- Lag: Current lag count
- Shows connected source connector with type, name, and status
- Shows connected destination connector with type, name, and status
- Click on connector cards to navigate to their detail pages
- Name: Topic name (clickable to view topic details)
- Type:
- Topic: Standard Kafka topic from source tables
- Transform: Topic created by a transform operation
- Actions: Action menu for each topic
Settings Tab
The Settings tab allows you to configure pipeline behavior and data selection:
- Edit the pipeline name
- Add or remove tags
- Create new tags or select from existing ones
- Tags support filtering and organization
- Toggle to enable periodic auditing of data consistency
- Helps detect and alert on data discrepancies between source and destination
- Select which databases and tables to replicate
- Search functionality to filter databases and tables
- Expandable tree view showing:
- Database level with checkbox to select all tables
- Individual tables with individual checkboxes
- “Select all” option to include all databases and tables
- Select specific transformed topics to include in the pipeline
- Hierarchical view showing schemas and topics
- Mixed checkbox states indicate partial selections
Changes in the Settings tab require saving before they take effect. Click the Save button at the bottom right to apply your changes.
Source and Destination Links
The pipeline detail page includes direct links to:- Source tab/link: Opens the source connector detail page in a new context
- Destination tab/link: Opens the destination connector detail page in a new context
Pagination
At the bottom of the pipelines table:- View the range of pipelines currently displayed (e.g., “Showing pipelines 1 to 10 of 20”)
- Change the number of pipelines per page (10, 25, 50, 100)
- Navigate between pages using Previous/Next buttons or page numbers
Performance Metrics
Latency
Latency measures how long it takes for data to flow from the source through the pipeline to the destination. Lower latency means near real-time replication.- 0ms - 100ms: Excellent, near real-time
- 100ms - 1s: Good, minimal delay
- >1s: May indicate performance issues or heavy load
Lag
Lag represents the number of records or events that are waiting to be processed. Zero lag means the pipeline is fully caught up with the source.- 0: Fully caught up
- >0: Pipeline is behind, processing backlog
Monitor latency and lag regularly. Increasing values may indicate:
- Source database load is too high
- Destination connector is slow to write
- Network issues between components
- Need for performance optimization
Increased latency and/or lab may be expected when snapshots are running. Backfilling usually produces a much bigger load than normal CDC streaming, but the load is temporary while backfills are running.
Bulk Actions
Select multiple pipelines using the checkboxes in the Name column to perform bulk actions:- Start multiple pipelines at once (coming soon)
- Stop multiple pipelines at once (coming soon)
- Apply tags to multiple pipelines
- Delete multiple pipelines - with confirmation (coming soon)
Related Documentation
- Sources - Configure source database connectors
- Destinations - Configure destination warehouse connectors
- Transforms - Apply data transformations
- Topics - Manage Kafka topics
- Alerts - Monitor pipeline health and receive notifications
- Logs - View detailed pipeline logs for troubleshooting