Cloud Architecture
The architecture fundamentally contains a a control plane and a data plane.Control Plane
The control plane is a separate application that manages your deployment, carries out operations and monitors your services. This is hosted within Streamkap. Typical actions include- Adding, editing connectors
- Monitoring
- Setting up alerts
- Team management
- SSO
Data Plane
The data plane is where your data moves through. This is a fully managed Kubernetes deployment within a sub account/customer VPC. There are no machines, OS, patching to look after since our agent manages it all. The control plane will communicate with the data plane to set any configuration such as creating connectors as well as collect log and metric data, allowing 24/7 monitoring/support.
How much does Streamkap manage?
Streamkap will manage the entire Kubernetes environment, including deployment, monitoring, patching etc.How does the install happen?
All that is needed is a single cloud principal that grants initial provisioning access and this setup can done via any of these methods- 1-click install template
- Manually using the Cloud console
- Create base layer sandbox — VPC, Kubernetes cluster, ECR resources
- Install the runner/agent
- Capturing logs when requested + providing basic debugging capabilities
- Continuous monitoring of the install
- Managing components — provisioning terraform, helm, container components
- Syncing and managing OCI artifacts into the account
Setup
Node & Software Requirements
You do not need to provision these or install any software. Our installer will automatically deploy and maintain them, whether that is updating own software and the OS. A general deployment will require:- 4 nodes (2 CPU, 8GB)
- 2 nodes (4 CPU, 16GB)
Network Requirements
Between Control Plane and Data Plane
- VPN will be setup between us to secure the communication
- Port 443 (HTTPS)
- Port 9094,9095,9096 (Kafka)
- Port 443 (HTTPS)
- Port 9701 (Runner/Agent)
Communication between the sub-account and the sources/destinations
The sub-account needs to be able to communicate with all the connectors and resolve any internal hostname. The sub account will initiate the connection.FAQ
How Do I Deploy BYOC?
How Do I Deploy BYOC?
The normal deployment here is to create a sub-account in your cloud vendor for Streamkap to use. There are two deployment options within this:
- We create a full new install (including cluster + vpc)
- The customer creates the VPC
What resources will be deployed in the sub-account?
What resources will be deployed in the sub-account?
A Kubernetes cluster will be deployed which will contain a number of VMs.
The CPU count for the deployment depends on the volume of data being processed but 4 CPU/12GB setup is typical.
We mostly utilise object storage.
The CPU count for the deployment depends on the volume of data being processed but 4 CPU/12GB setup is typical.
We mostly utilise object storage.
Do I need to maintain any software?
Do I need to maintain any software?
No. Our agent/runner will maintain all systems and software in the sub-account.
We will deploy Kubernetes and our agent will maintain, upgrade and handle entirely.
We will deploy Kubernetes and our agent will maintain, upgrade and handle entirely.
Can we verify the software is secure?
Can we verify the software is secure?
All the images will be synced to the customer account and so you can enable your standard security measures in this account including image scanning.
Does my data leave my VPC/Data Plane
Does my data leave my VPC/Data Plane
Your confidential data does not leave the data plane.Monitoring logs and some configuration is stored with Streamkap.
This is all kept encrypted and covered under our SOC2 compliance.
This is all kept encrypted and covered under our SOC2 compliance.
How does Streamkap ensure the health of your BYOC deployment?
How does Streamkap ensure the health of your BYOC deployment?
We install collector agents to gather metrics, logs that will be sent back to the Streamkap control plane.
It will detect if any issues happen as well as attempt a remedy.
Failing this, our support team will be notified.
It will detect if any issues happen as well as attempt a remedy.
Failing this, our support team will be notified.
What data do we collect for monitoring?
What data do we collect for monitoring?
It is only metric and log related to the Kubernetes cluster and our software services.
Are there ingress/egress fees?
Are there ingress/egress fees?
The deployment should be placed within the same region and ideally zone as your current systems.Ingress is typically free regardless of the data source.Egress depends on where you are sending the data.
Egress is usually free in the same region/zone and only becomes more substantial if you are sending to a different cloud vendor — in which case it could cost up to $0.10 per GB.By different cloud vendor, this refers to AWS, GCP, Azure or Oracle Cloud for example.
Egress is usually free in the same region/zone and only becomes more substantial if you are sending to a different cloud vendor — in which case it could cost up to $0.10 per GB.By different cloud vendor, this refers to AWS, GCP, Azure or Oracle Cloud for example.