With the Bring Your Own Cloud option, Streakmap will host the control plane in our environment while the data plane is deployed within your VPC on your infrastructure in a sub-account of your cloud vendor.

We suppport AWS and Azure.

Cloud Architecture

The architecture fundamentally contains a a control plane and a data plane.

Control Plane

The control plane is a separate application that manages your deployment, carries out operations and monitors your services. This is hosted within Streamkap. Typical actions include

Adding, editing connectors
Monitoring
Setting up alerts
Team management
SSO

Data Plane

The data plane is where your data moves through. This is a fully managed Kubernetes deployment within a sub account/customer VPC. There are no machines, OS, patching to look after since our agent manages it all.

The control plane will communicate with the data plane to set any configuration such as creating connectors as well as collect log and metric data, allowing 24/7 monitoring/support.

How much does Streamkap manage?

Streamkap will manage the entire Kubernetes environment, including deployment, monitoring, patching etc.

How does the install happen?

All that is needed is a single cloud IAM account that grants initial provisioning access and this setup can done via any of these methods

1-click install template
Terraform
Manually using the Cloud console

This IAM user is only used during the initial install, and can be deleted after. It has create but no get access.

Streamkap uses this install role to do the initial install provisioning:

Create base layer sandbox — VPC, Kubernetes cluster, ECR resources
Install the runner/agent

From there, the runner takes over the install and no cross account access is ever required again, until deleting the install. The runner is responsible for the following:

Capturing logs when requested + providing basic debugging capabilities
Continuous monitoring of the install
Managing components — provisioning terraform, helm, container components
Syncing and managing OCI artifacts into the account

The runner will provision servers and install the data plane portion of the Streamkap application.

The runner is also responsible for the delivery of upgrades.

Setup

Node & Software Requirements

You do not need to provision these or install any software. Our installer will automatically deploy and maintain them, whether that is updating own software and the OS

4 x 2 CPU, 8GB
2 x 4 CPU, 16GB

Network Requirements

Between Control Plane and Data Plane

VPN will be setup between us to secure the communication

Inbound from Streamkap Control Plane to Customer Data Plane

Port 443 (Kafka Connect, Schema Registry and Flink)
Port 9094 (Kafka)

Outbound from Customer Data Plane to Streamkap Control Plane

Port 8433 (Logs)
Port 9090 (Monitoring)
Port 443 (https)
Port 9701 (Runner/Agent)

Communication between the sub-account and the sources/destinations

The sub-account needs to be able to communicate with all the connectors. The sub account will initiate the connection.

FAQ

How Do I Deploy BYOC?

The normal deployment here is to create a sub-account in your cloud vendor for Streamkap to use. There are two deployment options within this

We create a full new install (including cluster + vpc)
The customer creates the VPC

What resources will be deployed in the sub-account?

A kubernetes cluster will be deployed which will contain a number of VMs. The CPU count for the deployment depends on the volume of data being processed but 4 CPU/12GB setup is typical. We mostly utilise object storage.

Do I need to maintain any software?

No. Our agent/runner will maintain all systems and software in the sub-account. We will deploy Kubernetes and our agent will maintain, upgrade and handle entirely.

Can we verify the software is secure?

All the images will be synced to the customer account and so you can enable your standard security measures in this account including image scanning.

Does my data leave my VPC/Data Plane

Your confidential data does not leave the data plane.

Monitoring logs and some configuration is stored with Streamkap. This is all kept encrypted and covered under our SOC2 compliance.

How streamkap ensures the health of your BYOC deployment?

We install collector agents to gather metrics, logs that will be sent back to streamkap control plane and it will detect if any issues happen as well as attempt a remedy. Failing this, our support team will be notified.

What data do we collect for monitoring?

It is only metric and log related related to the kuberentes cluster and our software services.

Are there ingress/egress fees?

The deployment should be placed within the same region and ideally zone as your current systems.

Ingress is typically free regardless of the data source.

Egress depends on where you are sending the data. Egress is usually free in the same region/zone and only becomes more substantial if you are sending to do a different cloud vendor - in which case it could cost up to $0.10 per GB. By different cloud vendor, this refers to AWS, GCP, Azure or Oracle Cloud for example.