📘

Available only on Enterprise Plan

With the Bring Your Own Cloud option, Streakmap will host the control plane in our environment while the data plane is deployed within your VPC on your infrastructure in a sub-account of your cloud vendor.

Cloud Architecture

The architecture fundamentally contains a a control plane and a data plane.

Control Plane

The control plane is a separate application that manages your deployment, carries out operations and monitors your services. Typical actions include

• Adding, editing connectors
• Monitoring
• Setting up alerts
• Team management
• SSO

Data Plane

The data plane is where your data moves through. The control plane will communicate with the data plane to set any configuration only and collect log and metric data.

How much does Streamkap manage?

Streamkap will manage the entire Kubernetes environment, including deployment, monitoring, patching etc.

How does the install happen?

All that is needed is a single cloud IAM account that grants initial provisioning access and this setup can done via any of these methods

• 1-click install template
• Terraform
• Manually using the Cloud console

This IAM user is only used during the initial install, and can be deleted after. It has create but no get access.

Streamkap uses this install role to do the initial install provisioning:

• Create base layer sandbox — VPC, Kubernetes cluster, ECR resources
• Install the runner/agent

From there, the runner takes over the install and no cross account access is ever required again, until deleting the install. The runner is responsible for the following:

• Capturing logs when requested + providing basic debugging capabilities
• Continuous monitoring of the install
• Managing components — provisioning terraform, helm, container components
• Syncing and managing OCI artifacts into the account

The runner will provision servers and install the data plane portion of the Streamkap application.

The runner is also responsible for the delivery of upgrades.

Setup

Network Requirements

Between Control Plane and Data Plane

• VPN will be setup between us to secure the communication

Inbound from Streamkap Control Plane to Customer Data Plane

• Port 443 (Kafka Connect, Schema Registry and Flink)
• Port 9094 (Kafka)

Outbound from Customer Data Plane to Streamkap Control Plane

• Port 8433 (Logs)
• Port 9090 (Monitoring)
• Port 443 (https)
• Port 9701 (Runner/Agent)

Communication between the sub-account and the sources/destinations

The sub-account needs to be able to communicate with all the connectors. The sub account will initiate the connection.

FAQ

How Do I Deploy BYOC?

The normal deployment here is to create a sub-account in your cloud vendor for Streamkap to use. There are two deployment options within this

1. We create a full new install (including cluster + vpc)
2. The customer creates the VPC

What resources will be deployed in the sub-account?

A kubernetes cluster will be deployed which will contain a number of VMs. The CPU count for the deployment depends on the volume of data being processed but 4 CPU/12GB setup is typical. We mostly utilise object storage.

Do I need to maintain any software?

No. Our agent/runner will maintain all systems and software in the sub-account. We will deploy Kubernetes and our agent will maintain, upgrade and handle entirely.

Can we verify the software is secure?

All the images will be synced to the customer account and so you can enable your standard security measures in this account including image scanning.

Does my data leave my VPC/Data Plane

Your confidential data does not leave the data plane.

Monitoring logs and some configuration is stored with Streamkap. This is all kept encrypted and covered under our SOC2 compliance.

Are there ingress/egress fees?

The deployment should be placed within the same region and ideally zone as your current systems.

Ingress is typically free regardless of the data source.

Egress depends on where you are sending the data. Egress is usually free in the same region/zone and only becomes more substantial if you are sending to do a different cloud vendor - in which case it could cost up to $0.10 per GB. By different cloud vendor, this refers to AWS, GCP, Azure or Oracle Cloud for example.