Connection Options

Available Connection Methods

  • Directly whitelisting our our IP addresses
  • SSH Tunnel
  • Reverse SSH Tunnel
  • AWS PrivateLink (Also can be used for On-Premise)
  • VPN
  • Proxy Agent

IP Whitelisting

Our IP addresses are here Streamkap IP Addresses

SSH Tunnel

  1. Setup SSH Host - Speak to your DevOps on this
  2. Create Group Streamkap
  • sudo groupadd streamkap
  1. Create user streamkap:
  • sudo useradd -m -g streamkap streamkap
  1. Switch to the streamkap user:
  • sudo su - streamkap
  1. Create the .ssh directory:
  • mkdir ~/.ssh
  1. Set permissions:
  • chmod 700 ~/.ssh
  1. Change to the .ssh directory:
  • cd ~/.ssh
  1. Create the authorized_keys file:
  • touch authorized_keys
  1. Set permissions:
  • chmod 600 authorized_keys
  1. Obtain the SSH key from Streamkap and add to your authorized_keys file. Ensure the key is one continuous line and no line breaks.
  2. Ensure there is access enabled from Streamkap IP addresses to your SSH Tunnel port and from your SSH Tunnel port to your database port.

AWS PrivateLink

AWS PrivateLink allows two AWS environments to connect while routing traffic within AWS. This prevents traffic being exposed to the internet while also reducing egress costs. Learn more about AWS PrivateLink

AWS PrivateLink can also be used to access on-premise databases by also utilising AWS Direct Connect. Learn more about AWS Direct Connect

AWS PrivateLink can be used for all sources within AWS as well as Snowflake & Databricks as a destination

Hosted database within AWS

📘

The flow of data here is Customer Database > Network Load Balancer (NLB) > Customer AWS Private Link > Streamkap AWS PrivateLink > Strreamkap. You will need to create an NLB inside your VPC if you do not already have one

  1. Create Network Load Balancer (NLB) within your VPC and configure it for each subnet/availability zone if it does not exist already
  2. Create a VPC endpoint service and choose your Network Load Balancer (NLB)
  3. Safelist Streamkap's AWS VPC Account ID (arn:aws:iam::300973880807:root). This will allow inbound access from our AWS PrivateLink
  4. Send the service name generated here to Streamkap so that we setup our AWS PrivateLink to talk to yours.
  5. Update Acceptance Settings to your preference. If you disable it, Streamkap will be able to connect automatically without permission. You have already safelisted our account in earlier steps

Network Load Balancers (NLB) can route traffic to an EC2 instance, an IP address, or a Lambda function through target groups.

There are two ways to set up an NLB to send traffic towards your RDS database: using port forwarding or using the RDS IP address. Below is how you do both:

Using a port forwarding instance

You must deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS database). Here is a sample script that you can use to set up the EC2 port forwarding instance:

# !/bin/bash
PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart.  
if test -f "$PREVLOGFILE"; then  
  truncate -s 0 $PREVLOGFILE  
  echo "State file $PREVLOGFILE has been emptied"  
fi  
python -m SimpleHTTPServer 801 & # NOTE: THIS PORT MUST MATCH THE...  
echo 1 -> /proc/sys/net/ipv4/ip_forward  
export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE \<\<>> CHARACTERS, NO QUOTATION MARKS.  
export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE \<\<>> CHARACTERS, NO QUOTATION MARKS.  
iptables -t nat -A POSTROUTING -j MASQUERADE  
while true  
do  
LOGFILE=/root/ip.txt  
Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked.  
if [ $LOGFILE = "" ] ; then  
  iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT  
  echo $Current_IP > $LOGFILE  
else  
  Old_IP=$(cat $LOGFILE)  
  if [ "$Current_IP" = "$Old_IP" ] ; then  
    echo "IP address has not changed ($Old_IP -> $Current_IP)"  
  else  
    iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT  
    iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT  
    sysctl net.ipv4.ip_forward=1  
    iptables-save  
    echo $Current_IP > $LOGFILE  
    echo "IP address has changed ($Old_IP -> $Current_IP)"  
  fi  
fi  
sleep 5  
done

After you create the port forwarding instance, set up the NLB listener and target group to lead traffic to the portforwarder EC2 instance.

Using an RDS IP address

RDS gives you an endpoint to access your database when you set up RDS. This endpoint resolves to an IP address. However, AWS suggests that instead of using this always-changing IP address, you can deploy a lambda function check the current IP address and update the NLB target group accordingly.

To use the RDS IP address in your NLB target group, do the following:

  1. Run nslookup/dig with the domain name of RDS endpoint as the input to find the IP address:
  • nslookup DNS_ENDPOINT
  • dig +short DNS_ENDPOINT
  1. Configure your NLB target group with the IP address from above
  2. Deploy a lambda function to periodically perform nslookup on the RDS endpoint to see if the IP address has changed and update the target group with the new IP address.

VPN

Please contact your account manager to setup a VPN connection.

Proxy Agent

The Proxy Agent is used to read data from on-prem environments and send the data to Streamkap.

We support this via Docker to ensure a better experience with the compatibility of hosts and updates we provide.

Platform

We use Docker Engine to allow for supporting multiple platforms. Please refer to Docker Engine minimum requirements for both hardware/software.

Streamkap Component

The sizing will depend on the volume of data being processed as well as whether you wish to run in a distributed state to support high availability. For high availability, add additional nodes of the same spec.

Streamkap requires:

Testing: 2 CPU, 4GB memory, Storage 4GB x 1 node

Production: 2 CPU, 4GB memory, Storage 4GB x 2 nodes

Network

  • Typically used with Network load balancer or Site-to-Site VPN. IP address whitelisting over the tunnel to be agreed at setup.
  • Open connection access from the Proxy Agent to the sources. This will be the standard database address and ports for each database.
  • Open connection access between the Proxy Agent & Streamkap
    • Inbound from Streamkap to Proxy Agent on Port 8083
    • Outbound Proxy Agent to Streamkap on Port 8081, 9095, 9096, 9097, 9131
    • Outbound Proxy Agent to Streamkap Metrics on Port 8443 - URL will be given to you during installation.

Installation

  • We will send you a deploy.sh script. This will
  1. Install Docker engine if it does not exist
  2. Configure log rotation to 1G
  3. Pull images and spin up there containers with automatic restart

Updated 1 day ago