API Sources
How Streamkap's API sources work, how they differ from CDC sources, and how they deliver to a destination
An API source pulls records out of a SaaS application by calling its REST API on a schedule, and writes them to Kafka topics — one topic per object you select. HubSpot, Salesforce and NetSuite are API sources.
You create one from Add Connectors like any other source, and it lists on the Sources page under the API tab, beside Streaming.
How an API source differs from a CDC source
A CDC source reads a database’s change log. Every insert, update and delete is written to that log by the database itself, so Streamkap sees each change once, in order, moments after it is committed.
A vendor’s REST API has no change log to read. An API source therefore polls: it asks the vendor for records that changed since the last poll, and repeats on a cadence. That difference shows up in a few places you will notice:
| CDC source | API source | |
|---|---|---|
| How changes arrive | Streamed from the database’s change log | Polled from the vendor’s REST API on a cadence |
| Latency | Near real-time | One poll interval |
| What you select | Databases, schemas and tables | Objects or resources the vendor exposes |
| Deletes | Read from the change log | Detected by a periodic sweep, so they arrive later than inserts and updates |
| Limits to respect | Your database’s load | The vendor’s API rate limits and daily quotas |
| Schema tab | Yes | No — there is no table schema to include or exclude |
What is the same: records land flat in your destination like any other Streamkap source, deletes arrive as __deleted = true tombstones, and every existing destination consumes them unchanged.
API sources do not use pipelines
API sources do not appear in the Create pipeline source picker. This is deliberate, not a bug or a missing permission. If you went to Pipelines → Create looking for HubSpot, Salesforce or NetSuite and could not find it, this is why.
An API source is linked to a destination with Send to destination instead. You pick topics and a destination, and Streamkap creates and maintains the pipeline for you. It appears on the Pipelines page with an Auto-managed chip that links back to the source’s Destinations tab, which is the one place it is edited or removed.
Where to find Send to destination
You can start from whichever end you are already looking at:
- From the source — open it, go to the Destinations tab, and click Send topics. Choose all of the source’s topics or pick individual ones, then pick a destination.
- From the Topics page — select the topics you want with their checkboxes and click Send to destination in the bulk actions bar.
- From the destination — open it and click Add topics next to the Topics heading.
Before you commit, the picker tells you exactly what will happen — how many topics go to which destination, and whether any of them already deliver there.
To stop delivering, go back to the source’s Destinations tab: remove individual topics, or use Detach all to clear a destination. Removing the last topic removes the destination from the source entirely.
The lifecycle of an API source
Every API source follows the same four steps, whatever the vendor:
Connect your credentials
Create the source from Add Connectors and authenticate on the Auth step — a token you paste, or an authorization you grant in the vendor’s own consent screen. Credentials are stored encrypted and never shown again.
Test the connection
Click Test connection. It runs a read-only check — nothing is created or written — and reports whether the credentials work and, where the vendor supports it, whether each object you selected can actually be read. It also warns you if the vendor account is already connected to another source in this workspace, which is not allowed.
Pick the objects to sync
On the Settings step, choose the objects or resources you want. Each one becomes its own Kafka topic, named source_<id>.<vendor>.<resource>; the panel beside the form previews them. Optionally set a Backfill start date to bound how far back the first sync reads.
Send the topics to a destination
Once the source is running, use Send to destination to deliver its topics. Until you do, the source streams to its topics but nothing downstream receives them — the source’s Status tab says so, with a shortcut to fix it.
What every API source does
These behaviors are shared, and each connector page documents its specifics:
- First sync reads in full, then goes incremental. Each object is read completely on its first poll; after that only records that changed since the last poll are fetched. A poll where nothing changed produces zero records — that is expected, not a fault.
- Deletes arrive on a sweep. A periodic sweep detects deleted or archived records and emits
__deleted = true, so your destination flips the row to deleted. Expect deletes to lag inserts and updates by up to one sweep interval. - Rate limits are respected. The connector paces itself below the vendor’s limits so it leaves headroom for your other integrations. If a daily quota is exhausted anyway, ingestion pauses and resumes without losing data.
- Permission problems are per object where the vendor allows it. An object you lack access to can pause on its own while the rest keep streaming, and recovers once you grant access.
- New fields are picked up automatically. The field list is read from your account, not hardcoded, so a property you add appears without reconfiguring Streamkap. Your destination adds the column if it is set to evolve its schema — see Schema Evolution.
Available API sources
HubSpot
CRM objects — contacts, deals, companies, tickets and more — via a Private App access token.
Salesforce
Standard and custom objects, with a service account or by connecting with Salesforce over OAuth.
NetSuite
Standard and custom records over SuiteQL, with a certificate or an existing token-based integration.
Using the API
API sources are created and updated through the same POST /sources and PUT /sources/{source_id} endpoints as any other source. Linking them to a destination is the one thing that differs: use the topic-destinations endpoints rather than POST /pipelines, which rejects an API source. See Send an API Source to a Destination.