Connection Options
Available Connection Methods
- Directly whitelisting our our IP addresses
- SSH Tunnel
- Reverse SSH Tunnel
- AWS PrivateLink (Also can be used for On-Premise)
- VPN
- Proxy Agent
IP Whitelisting
Our IP addresses are here Streamkap IP Addresses
SSH Tunnel
- Setup SSH Host - Speak to your DevOps on this
- Create Group Streamkap
sudo groupadd streamkap
- Create user streamkap:
sudo useradd -m -g streamkap streamkap
- Switch to the streamkap user:
sudo su - streamkap
- Create the .ssh directory:
mkdir ~/.ssh
- Set permissions:
chmod 700 ~/.ssh
- Change to the .ssh directory:
cd ~/.ssh
- Create the authorized_keys file:
touch authorized_keys
- Set permissions:
chmod 600 authorized_keys
- Obtain the SSH key from Streamkap and add to your authorized_keys file. Ensure the key is one continuous line and no line breaks.
- Ensure there is access enabled from Streamkap IP addresses to your SSH Tunnel port and from your SSH Tunnel port to your database port.
AWS PrivateLink
AWS PrivateLink allows two AWS environments to connect while routing traffic within AWS. This prevents traffic being exposed to the internet while also reducing egress costs. Learn more about AWS PrivateLink
AWS PrivateLink can also be used to access on-premise databases by also utilising AWS Direct Connect. Learn more about AWS Direct Connect
AWS PrivateLink can be used for all sources within AWS as well as Snowflake & Databricks as a destination
Hosted database within AWS
The flow of data here is Customer Database > Network Load Balancer (NLB) > Customer AWS Private Link > Streamkap AWS PrivateLink > Strreamkap. You will need to create an NLB inside your VPC if you do not already have one
- Create Network Load Balancer (NLB) within your VPC and configure it for each subnet/availability zone if it does not exist already
- Create a VPC endpoint service and choose your Network Load Balancer (NLB)
- Safelist Streamkap's AWS VPC Account ID (arn:aws:iam::300973880807:root). This will allow inbound access from our AWS PrivateLink
- Send the service name generated here to Streamkap so that we setup our AWS PrivateLink to talk to yours.
- Update Acceptance Settings to your preference. If you disable it, Streamkap will be able to connect automatically without permission. You have already safelisted our account in earlier steps
Network Load Balancers (NLB) can route traffic to an EC2 instance, an IP address, or a Lambda function through target groups.
There are two ways to set up an NLB to send traffic towards your RDS database: using port forwarding or using the RDS IP address. Below is how you do both:
Using a port forwarding instance
You must deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS database). Here is a sample script that you can use to set up the EC2 port forwarding instance:
# !/bin/bash
PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart.
if test -f "$PREVLOGFILE"; then
truncate -s 0 $PREVLOGFILE
echo "State file $PREVLOGFILE has been emptied"
fi
python -m SimpleHTTPServer 801 & # NOTE: THIS PORT MUST MATCH THE...
echo 1 -> /proc/sys/net/ipv4/ip_forward
export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE \<\<>> CHARACTERS, NO QUOTATION MARKS.
export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE \<\<>> CHARACTERS, NO QUOTATION MARKS.
iptables -t nat -A POSTROUTING -j MASQUERADE
while true
do
LOGFILE=/root/ip.txt
Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked.
if [ $LOGFILE = "" ] ; then
iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT
echo $Current_IP > $LOGFILE
else
Old_IP=$(cat $LOGFILE)
if [ "$Current_IP" = "$Old_IP" ] ; then
echo "IP address has not changed ($Old_IP -> $Current_IP)"
else
iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT
sysctl net.ipv4.ip_forward=1
iptables-save
echo $Current_IP > $LOGFILE
echo "IP address has changed ($Old_IP -> $Current_IP)"
fi
fi
sleep 5
done
After you create the port forwarding instance, set up the NLB listener and target group to lead traffic to the portforwarder EC2 instance.
Using an RDS IP address
RDS gives you an endpoint to access your database when you set up RDS. This endpoint resolves to an IP address. However, AWS suggests that instead of using this always-changing IP address, you can deploy a lambda function check the current IP address and update the NLB target group accordingly.
To use the RDS IP address in your NLB target group, do the following:
- Run nslookup/dig with the domain name of RDS endpoint as the input to find the IP address:
- nslookup DNS_ENDPOINT
- dig +short DNS_ENDPOINT
- Configure your NLB target group with the IP address from above
- Deploy a lambda function to periodically perform nslookup on the RDS endpoint to see if the IP address has changed and update the target group with the new IP address.
VPN
Please contact your account manager to setup a VPN connection.
Proxy Agent
The Proxy Agent is used to read data from on-prem environments and send the data to Streamkap.
We support this via Docker to ensure a better experience with the compatibility of hosts and updates we provide.
Platform
We use Docker Engine to allow for supporting multiple platforms. Please refer to Docker Engine minimum requirements for both hardware/software.
Streamkap Component
The sizing will depend on the volume of data being processed as well as whether you wish to run in a distributed state to support high availability. For high availability, add additional nodes of the same spec.
Streamkap requires:
Testing: 2 CPU, 4GB memory, Storage 4GB x 1 node
Production: 2 CPU, 4GB memory, Storage 4GB x 2 nodes
Network
- Typically used with Network load balancer or Site-to-Site VPN. IP address whitelisting over the tunnel to be agreed at setup.
- Open connection access from the Proxy Agent to the sources. This will be the standard database address and ports for each database.
- Open connection access between the Proxy Agent & Streamkap
- Inbound from Streamkap to Proxy Agent on Port 8083
- Outbound Proxy Agent to Streamkap on Port 8081, 9095, 9096, 9097, 9131
- Outbound Proxy Agent to Streamkap Metrics on Port 8443 - URL will be given to you during installation.
Installation
- We will send you a deploy.sh script. This will
- Install Docker engine if it does not exist
- Configure log rotation to 1G
- Pull images and spin up there containers with automatic restart
Updated 1 day ago