Starburst
Starburst Change Data Capture with Streamkap
Requirements
- AWS Access Key and Secret Access Key with the following permissions to the destination bucket:
s3:GetObject
s3:PutObject
s3:AbortMultipartUpload
s3:ListMultipartUploadParts
s3:ListBucketMultipartUploads
Configure S3 Connector
Name
: A descriptive name for the connectorAWS Access Key
: An Access Key with the appropriate permissions for the bucket to which Streamkap will load dataAWS Secret Access Key
: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load dataRegion
: Name of the region for bucket to which Streamkap will load dataBucket Name
: The name of the bucket to which Streamkap will load dataFormat
: The format of the file. The following options are available:Avro
,Byte Array
,JSON
andParquet
Directory
: The name of the directory to which Streamkap will load data
If the Format Type selected is JSON
or Byte Array
:
Compression Type
: How Streamkap should compress the files. The following options are available:None
orgzip
If the Format Type selected is Avro
:
Avro Compression Type
: How Streamkap should compress the files. The following options are available:null
,deflate
,snappy
, orbzip2
If the Format Type selected is Parquet
:
Parquet Compression Type
: How Streamkap should compress the files. The following options are available:None
,gzip
,snappy
,lz4
,brotli
,zstd
, orlzo
Partition Field
: How the files should be partitioned.Automatic
will partition the data automatically and not based on any particular field.Field
will partition the data by the values of a specified field.Time
will partition the data by the time of a particular field
If the Partition Field
is set to Time
:
Time Partition Directory Path Format
: The format of the partitioned directories. For example, if you set format to'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH
, the data directories will have the format/year=2015/month=12/day=07/hour=15/
Time Partition Timezone
: The timezone of the partitioned directories. Accepts short and long standard names like:UTC
,PST
,ECT
,Europe/Berlin
,Europe/Helsinki
, orAmerica/New_York
Updated 12 months ago