Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Elasticsearch

Configure the Elasticsearch source connector in Streamkap, including cluster connection details, authentication, and index capture settings.

Prerequisites

  • Elasticsearch cluster connection details
  • A database user with sufficient privileges to read from indices

Elasticsearch Setup

1. Grant Database Access

  • Configure one of the Connection Options to ensure Streamkap can reach your Elasticsearch cluster.

2. Create Database User

  • Create a user and password for Streamkap with read access to the indices you want to capture.

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Source

2. Connection Settings

  • Name: Enter a name for your connector.

  • Hostname: Specify the Elasticsearch cluster hostname.

  • Port: Default is 443 for HTTPS.

  • Protocol: Select HTTPS or HTTP.

  • Authentication Type: Select the authentication method.

    • For Basic authentication:
      • Username: The Elasticsearch user.
      • Password: The user’s password.

3. Index Capture

  • Index names: Specify the index(es) to capture.

  • Datetime Field name: The field used to track changes (e.g. @timestamp).

  • Tiebreak Field name: A second field used to put records in order when several share the same datetime value. Found under Advanced.

Click Save.

Why the Tiebreak Field matters

Streamkap reads your index in datetime order, a page of records at a time. After each page it remembers the datetime of the last record it read, and next time asks Elasticsearch for everything that comes after it.

This works as long as the datetime tells records apart. If more records share a single datetime value than fit in one page, the ones left over are never asked for again. They are missed silently - nothing fails and nothing is logged.

How likely this is depends on how precise your datetime field is. A field stored to the second is a common cause: a busy index can easily write more records in the same second than fit in one page.

The Tiebreak Field fixes this. Streamkap orders records by your datetime field and then by the tiebreak field, so every record has a unique position and none are skipped.

Choosing a Tiebreak Field

Pick a field that is:

  • Unique for every record in the index, such as an ID or an order number.
  • Sortable, meaning a keyword, number or date field.

If no field in your index is suitable, you can use _id, the identifier Elasticsearch gives every record. This works, but it uses more memory on your Elasticsearch cluster than a normal field, so prefer a field of your own where you have one.

Adding a Tiebreak Field to an existing connector

Existing connectors keep working exactly as before until you set this field. Turning it on is safe and does not re-read your history.

1. Choose the field

Use the guidance in Choosing a Tiebreak Field. Confirm the field exists in every index the connector captures, and that it is a keyword, number or date field rather than text.

2. Update the connector

  • Open the connector and select Edit.
  • Under Advanced, set Tiebreak Field name.
  • Save.

The connector restarts. It carries on from where it was, so there is no re-sync from the beginning.

3. What to expect afterwards

If duplicates are a problem for your destination, match on your record’s own unique field rather than counting rows.

4. Checking it worked

To confirm nothing is being missed, count records in Elasticsearch for a fixed datetime window and compare that to the number of distinct records that reached your destination for the same window. Count distinct rather than total, so the expected duplicates above are not mistaken for a problem.

Removing the setting

Clearing the field returns the connector to its previous behaviour, including the risk of missed records. Nothing else needs changing.