TypeScript Development Kit

Build Streamkap transforms using TypeScript with your favorite npm packages and full testing support. Use as a standalone project or integrate into your existing codebase. Repository: streamkap-com/transform-kit-typescript

Choose Your Path

🆕 New Project (Start from scratch)

Use the entire repository as your development environment

🔄 Existing Project (Add to your codebase)

Add TypeScript transform capabilities to your current project

Quick Start - New Project

Prerequisites: Node.js 16+, npm 8+

git clone https://github.com/streamkap-com/transform-kit-typescript.git
cd transform-kit-typescript
npm install && npm run build && npm test
ls -la transforms/  # ← Copy these .js files to Streamkap

Note: You may see some npm warnings during install - these don’t affect functionality. 📋 Next: See Which File to Use? and deployment steps below.

Supported Transform Types

Transform Type	Files Generated	Description
Map/Filter	`valueTransform.js`, `keyTransform.js`	Transform and filter records
Fan Out	`valueTransform.js`, `topicTransform.js`	Route records to multiple topics
Enrich (Async)	`valueTransform.js`	Async data enrichment
Un-nesting	`valueTransform.js`	Flatten nested objects

How to Use - New Project

1. Define Your Data Structure

Tell the system what your data looks like by editing the TypeScript files:

// src/OrderType1.ts
export interface OrderType1 {
    _id: string;
    order_number: number;
    total_amount: number;
}

// src/MergedOrder.ts
export interface MergedOrder {
    _id: string;
    processed_at: string;
    formatted_total: string;
}

2. Write Your Business Logic

Open src/OrderTransformer.ts and write the code that processes your data:

export class OrderTransformer {
    transform(input: OrderType1): MergedOrder {
        const moment = require('moment');
        
        return {
            _id: input._id,
            processed_at: moment().toISOString(),
            formatted_total: `$${input.total_amount.toFixed(2)}`
        };
    }
}

3. Add External Libraries (Optional)

You can use popular JavaScript libraries to help with your transforms:

npm install moment lodash uuid  # These are already installed in the project

What works: JavaScript libraries like moment (dates), lodash (utilities), uuid What doesn’t work: Libraries that require system-level access or Node.js-specific features

4. Build and Test

npm run build  # Generates JavaScript files
npm test       # Runs all tests

Files are generated in the transforms/ directory.

5. Deploy to Streamkap

Coming soon: deploy using CLI , deploy using Terraform.

Create your transform in the Streamkap web interface
Go to the Implementation tab
Copy the entire contents of your generated file:
- Always required: valueTransform.js (your main business logic)
- Optional: keyTransform.js (only if you need custom partitioning - leave blank otherwise)
- Optional: topicTransform.js (only for fan-out routing - leave blank otherwise)
Paste into Streamkap’s code editor (this replaces the default code)
Save and Deploy - your TypeScript code is now running!

The files are completely self-contained, so just copy the entire file content.

🤔 Which File to Use?

Transform Type	File to Copy	Purpose
map_filter	`valueTransform.js` (required)	Transform and filter records
	`keyTransform.js` (optional)	Leave blank in Streamkap if no key changes needed
fan_out	`valueTransform.js` (required)	Transform records for routing
	`topicTransform.js` (optional)	Leave blank in Streamkap if no topic changes needed
enrich_async	`valueTransform.js` (required)	Enrich with external APIs
un_nesting	`valueTransform.js` (required)	Flatten nested objects

💡 Key Points:

valueTransform.js is always required (contains your main business logic)
keyTransform.js is optional - only needed for custom partitioning logic; leave blank in Streamkap if you want to keep the original message key
topicTransform.js is optional - only needed for fan-out transforms to route messages to different topics; leave blank in Streamkap if sending to the original topic

Generated Files

transforms/
├── map-filter/
│   ├── valueTransform.js
│   └── keyTransform.js
├── fan-out/
│   ├── valueTransform.js
│   └── topicTransform.js
├── enrich-async/
│   └── valueTransform.js
└── un-nesting/
    └── valueTransform.js

Each file is self-contained with all npm dependencies bundled inside.

How to Use - Existing Project

Add Streamkap bundler to your existing codebase in 3 steps:

1. Copy Files

# Copy bundler and transform files
cp build-multiple.js your-project/
cp test-selective.js your-project/
mkdir -p your-project/src/
cp src/value_transform.ts your-project/src/
cp src/key_transform.ts your-project/src/
cp src/topic_transform.ts your-project/src/
# Optionally copy the example interfaces and transformer as reference
cp src/OrderTransformer.ts your-project/src/ 
cp src/Customer.ts your-project/src/
cp src/OrderType1.ts your-project/src/
cp src/OrderType2.ts your-project/src/
cp src/MergedOrder.ts your-project/src/

2. Update package.json

{
  "scripts": {
    "build": "node build-multiple.js --all",
    "build:map-filter": "node build-multiple.js --map-filter",
    "build:fan-out": "node build-multiple.js --fan-out",
    "build:enrich-async": "node build-multiple.js --enrich-async",
    "build:un-nesting": "node build-multiple.js --un-nesting",
    "test": "node test-selective.js",
    "test:map-filter": "npm run build:map-filter && node test-selective.js --map-filter"
  },
  "dependencies": {
    "lodash": "^4.17.21",
    "moment": "^2.30.1", 
    "uuid": "^11.1.0"
  },
  "devDependencies": {
    "@types/lodash": "^4.17.20",
    "@types/moment": "^2.11.29",
    "@types/uuid": "^10.0.0",
    "esbuild": "^0.8.27",
    "typescript": "^4.0.0"
  }
}

3. Connect Your Logic

Create your own OrderTransformer or update the existing one:

// src/YourTransformer.ts
import { YourService } from './your-existing-logic';

export class YourTransformer {
    private yourService = new YourService();
    
    public transform(input: any): any {
        // Use your existing business logic
        return this.yourService.processData(input);
    }
}

Then update the transform files to use your transformer:

// src/value_transform.ts
import { YourTransformer } from "./YourTransformer";

export function _streamkap_transform(valueObject: any, keyObject: any, topic: any, timestamp: any) {
    var transformer = new YourTransformer();
    var transformedRecord = transformer.transform(valueObject);
    return transformedRecord;
}

4. Build & Deploy

# Install dependencies
npm install

# Build all transforms
npm run build

# OR build specific transforms only
npm run build:map-filter                            # Build specific type
node build-multiple.js --map-filter --enrich-async  # Build multiple types
node build-multiple.js --all                        # Build all transforms

# Check generated files
ls transforms/  # Copy these .js files to Streamkap

That’s it! Your transforms are generated in transforms/ folder. 📋 Next: See Which File to Use? below for deployment guidance.

Real-World Examples

These are just examples - replace the data types and logic with your own use case.

Order Processing

transform(input: Order): EnrichedOrder {
    const moment = require('moment');
    
    return {
        _id: input._id,
        processed_at: moment().format(),
        priority: input.total > 100 ? 'high' : 'normal'
    };
}

Sensor Data

transform(input: SensorReading): ProcessedSensor {
    const _ = require('lodash');
    
    return {
        sensor_id: input.sensor_id,
        average: _.meanBy(input.readings, 'value'),
        count: input.readings.length
    };
}

Order Processing (Actual Implementation)

transformOrderType1(inputOrder: OrderType1): MergedOrder {
    const now = moment();
    
    // Example using lodash for data manipulation
    const cleanedOrder = _.omitBy(inputOrder, _.isUndefined);
    const hasValidCustomer = _.has(cleanedOrder, 'customer.name') && !_.isEmpty(cleanedOrder.customer.name);
    
    // Example using uuid for unique identifiers
    const processingId = uuidv4();
    
    return {
        version: '0.1.4',
        _id: inputOrder._id,
        order_number: inputOrder.order_number,
        processed_at: now.toISOString(),
        processed_time: now.format('YYYY-MM-DD HH:mm:ss'),
        processing_id: processingId, // uuid generated
        has_valid_customer: hasValidCustomer, // lodash validation
        field_count: _.keys(cleanedOrder).length, // lodash utility
        // ... other fields
    };
}

Key Files You’ll Work With

New Project:

src/
├── OrderTransformer.ts     # Your main business logic goes here
├── OrderType1.ts          # Define what your input data looks like
├── MergedOrder.ts         # Define what your output data looks like
├── *.test.ts             # Automated tests (optional to modify)
└── index.ts              # System file (don't modify)

Existing Project:

src/templates/
├── commonTransform.ts     # Connect your existing services here
├── valueTransform.ts      # Main value transformation logic
├── keyTransform.ts        # Key transformation for partitioning
├── topicTransform.ts      # Topic routing for fan-out
└── index.ts              # System file (don't modify)

Both approaches generate:

transforms/               # Generated files (copy these to Streamkap)

What You Get

Type Safety: Catch errors before deployment with TypeScript
Rich Ecosystem: Use thousands of npm packages in your transforms
Testing: Automated tests ensure your code works correctly
Self-Contained: Generated files include everything needed - no external dependencies
All Transform Types: Works with every Streamkap transform type
Two Integration Options: Use as standalone project or integrate into existing codebase
Standard Workflow: Familiar git-based development process

Troubleshooting

Tests failing?

npm run build  # Build first - tests validate generated files
npm test -- --verbose

Build errors?

npm install
npm run build 2>&1 | grep -i error

Only need specific transforms?

# Build individual transform types for faster development
npm run build:map-filter      # Only builds map/filter transforms
npm run test:enrich-async     # Builds and tests async enrichment

# Test without building (uses existing builds)
npm test -- --map-filter     # Test only map/filter transforms
npm test -- --fan-out --un-nesting  # Test multiple specific types

# Build multiple specific types
node build-multiple.js --fan-out --un-nesting

# Available types: --map-filter, --fan-out, --enrich-async, --un-nesting, --all

Streamkap deployment not working?

Make sure you copy the entire file contents (including all the bundled code)
Always run npm test locally before deploying
Check the Streamkap error console for specific error messages

Always run npm run build before npm test. The tests check the actual files that get deployed to Streamkap.

Need More Details?

This guide covers the basics. For advanced usage, troubleshooting, and detailed explanations, check the README.md file in the repository.

Getting Started

App

Troubleshooting

Deployment

Sources

Destinations

Transformation

Usage

Security

Developer Tools

Terraform

TypeScript Development Kit

Choose Your Path

🆕 New Project (Start from scratch)

🔄 Existing Project (Add to your codebase)

Quick Start - New Project

Supported Transform Types

How to Use - New Project

1. Define Your Data Structure

2. Write Your Business Logic

3. Add External Libraries (Optional)

4. Build and Test

5. Deploy to Streamkap

🤔 Which File to Use?

Generated Files

How to Use - Existing Project

1. Copy Files

2. Update package.json

3. Connect Your Logic

4. Build & Deploy

Real-World Examples

Order Processing

Sensor Data

Order Processing (Actual Implementation)

Key Files You’ll Work With

What You Get

Troubleshooting

Need More Details?

​Choose Your Path

​🆕 New Project (Start from scratch)

​🔄 Existing Project (Add to your codebase)

​Quick Start - New Project

​Supported Transform Types

​How to Use - New Project

​1. Define Your Data Structure

​2. Write Your Business Logic

​3. Add External Libraries (Optional)

​4. Build and Test

​5. Deploy to Streamkap

​🤔 Which File to Use?

​Generated Files

​How to Use - Existing Project

​1. Copy Files

​2. Update package.json

​3. Connect Your Logic

​4. Build & Deploy

​Real-World Examples

​Order Processing

​Sensor Data

​Order Processing (Actual Implementation)

​Key Files You’ll Work With

​What You Get

​Troubleshooting

​Need More Details?

Choose Your Path

🆕 New Project (Start from scratch)

🔄 Existing Project (Add to your codebase)

Quick Start - New Project

Supported Transform Types

How to Use - New Project

1. Define Your Data Structure

2. Write Your Business Logic

3. Add External Libraries (Optional)

4. Build and Test

5. Deploy to Streamkap

🤔 Which File to Use?

Generated Files

How to Use - Existing Project

1. Copy Files

2. Update package.json

3. Connect Your Logic

4. Build & Deploy

Real-World Examples

Order Processing

Sensor Data

Order Processing (Actual Implementation)

Key Files You’ll Work With

What You Get

Troubleshooting

Need More Details?