Skip to main content
Build Streamkap transforms using TypeScript with your favorite npm packages and full testing support. Use as a standalone project or integrate into your existing codebase. Repository: streamkap-com/transform-kit-typescript

​
Choose Your Path

​
🆕 New Project (Start from scratch)

Use the entire repository as your development environment

​
🔄 Existing Project (Add to your codebase)

Add TypeScript transform capabilities to your current project

​
Quick Start - New Project

Prerequisites: Node.js 16+, npm 8+ 
git clone https://github.com/streamkap-com/transform-kit-typescript.git
cd transform-kit-typescript
npm install && npm run build && npm test
ls -la transforms/  # ← Copy these .js files to Streamkap
Note: You may see some npm warnings during install - these don’t affect functionality. 📋 Next: See Which File to Use? and deployment steps below.

​
Supported Transform Types

Transform TypeFiles GeneratedDescription
Map/FiltervalueTransform.js, keyTransform.jsTransform and filter records
Fan OutvalueTransform.js, topicTransform.jsRoute records to multiple topics
Enrich (Async)valueTransform.jsAsync data enrichment
Un-nestingvalueTransform.jsFlatten nested objects

​
How to Use - New Project

​
1. Define Your Data Structure

Tell the system what your data looks like by editing the TypeScript files: 
// src/OrderType1.ts
export interface OrderType1 {
    _id: string;
    order_number: number;
    total_amount: number;
}

// src/MergedOrder.ts
export interface MergedOrder {
    _id: string;
    processed_at: string;
    formatted_total: string;
}

​
2. Write Your Business Logic

Open src/OrderTransformer.ts and write the code that processes your data: 
export class OrderTransformer {
    transform(input: OrderType1): MergedOrder {
        const moment = require('moment');
        
        return {
            _id: input._id,
            processed_at: moment().toISOString(),
            formatted_total: `$${input.total_amount.toFixed(2)}`
        };
    }
}

​
3. Add External Libraries (Optional)

You can use popular JavaScript libraries to help with your transforms: 
npm install moment lodash uuid  # These are already installed in the project
What works: JavaScript libraries like moment (dates), lodash (utilities), uuid What doesn’t work: Libraries that require system-level access or Node.js-specific features

​
4. Build and Test

npm run build  # Generates JavaScript files
npm test       # Runs all tests
Files are generated in the transforms/ directory.

​
5. Deploy to Streamkap

Coming soon: deploy using CLI , deploy using Terraform.
  1. Create your transform in the Streamkap web interface
  2. Go to the Implementation tab
  3. Copy the entire contents of your generated file:
    • Always required: valueTransform.js (your main business logic)
    • Optional: keyTransform.js (only if you need custom partitioning - leave blank otherwise)
    • Optional: topicTransform.js (only for fan-out routing - leave blank otherwise)
  4. Paste into Streamkap’s code editor (this replaces the default code)
  5. Save and Deploy - your TypeScript code is now running!
The files are completely self-contained, so just copy the entire file content.

​
🤔 Which File to Use?

Transform TypeFile to CopyPurpose
map_filtervalueTransform.js (required)Transform and filter records
keyTransform.js (optional)Leave blank in Streamkap if no key changes needed
fan_outvalueTransform.js (required)Transform records for routing
topicTransform.js (optional)Leave blank in Streamkap if no topic changes needed
enrich_asyncvalueTransform.js (required)Enrich with external APIs
un_nestingvalueTransform.js (required)Flatten nested objects
💡 Key Points:
  • valueTransform.js is always required (contains your main business logic)
  • keyTransform.js is optional - only needed for custom partitioning logic; leave blank in Streamkap if you want to keep the original message key
  • topicTransform.js is optional - only needed for fan-out transforms to route messages to different topics; leave blank in Streamkap if sending to the original topic

​
Generated Files

transforms/
├── map-filter/
│   ├── valueTransform.js
│   └── keyTransform.js
├── fan-out/
│   ├── valueTransform.js
│   └── topicTransform.js
├── enrich-async/
│   └── valueTransform.js
└── un-nesting/
    └── valueTransform.js
Each file is self-contained with all npm dependencies bundled inside.

​
How to Use - Existing Project

Add Streamkap bundler to your existing codebase in 3 steps:

​
1. Copy Files

# Copy bundler and transform files
cp build-multiple.js your-project/
cp test-selective.js your-project/
mkdir -p your-project/src/
cp src/value_transform.ts your-project/src/
cp src/key_transform.ts your-project/src/
cp src/topic_transform.ts your-project/src/
# Optionally copy the example interfaces and transformer as reference
cp src/OrderTransformer.ts your-project/src/ 
cp src/Customer.ts your-project/src/
cp src/OrderType1.ts your-project/src/
cp src/OrderType2.ts your-project/src/
cp src/MergedOrder.ts your-project/src/

​
2. Update package.json

{
  "scripts": {
    "build": "node build-multiple.js --all",
    "build:map-filter": "node build-multiple.js --map-filter",
    "build:fan-out": "node build-multiple.js --fan-out",
    "build:enrich-async": "node build-multiple.js --enrich-async",
    "build:un-nesting": "node build-multiple.js --un-nesting",
    "test": "node test-selective.js",
    "test:map-filter": "npm run build:map-filter && node test-selective.js --map-filter"
  },
  "dependencies": {
    "lodash": "^4.17.21",
    "moment": "^2.30.1", 
    "uuid": "^11.1.0"
  },
  "devDependencies": {
    "@types/lodash": "^4.17.20",
    "@types/moment": "^2.11.29",
    "@types/uuid": "^10.0.0",
    "esbuild": "^0.8.27",
    "typescript": "^4.0.0"
  }
}

​
3. Connect Your Logic

Create your own OrderTransformer or update the existing one: 
// src/YourTransformer.ts
import { YourService } from './your-existing-logic';

export class YourTransformer {
    private yourService = new YourService();
    
    public transform(input: any): any {
        // Use your existing business logic
        return this.yourService.processData(input);
    }
}
Then update the transform files to use your transformer: 
// src/value_transform.ts
import { YourTransformer } from "./YourTransformer";

export function _streamkap_transform(valueObject: any, keyObject: any, topic: any, timestamp: any) {
    var transformer = new YourTransformer();
    var transformedRecord = transformer.transform(valueObject);
    return transformedRecord;
}

​
4. Build & Deploy

# Install dependencies
npm install

# Build all transforms
npm run build

# OR build specific transforms only
npm run build:map-filter                            # Build specific type
node build-multiple.js --map-filter --enrich-async  # Build multiple types
node build-multiple.js --all                        # Build all transforms

# Check generated files
ls transforms/  # Copy these .js files to Streamkap
That’s it! Your transforms are generated in transforms/ folder. 📋 Next: See Which File to Use? below for deployment guidance.

​
Real-World Examples

These are just examples - replace the data types and logic with your own use case.

​
Order Processing

transform(input: Order): EnrichedOrder {
    const moment = require('moment');
    
    return {
        _id: input._id,
        processed_at: moment().format(),
        priority: input.total > 100 ? 'high' : 'normal'
    };
}

​
Sensor Data

transform(input: SensorReading): ProcessedSensor {
    const _ = require('lodash');
    
    return {
        sensor_id: input.sensor_id,
        average: _.meanBy(input.readings, 'value'),
        count: input.readings.length
    };
}

​
Order Processing (Actual Implementation)

transformOrderType1(inputOrder: OrderType1): MergedOrder {
    const now = moment();
    
    // Example using lodash for data manipulation
    const cleanedOrder = _.omitBy(inputOrder, _.isUndefined);
    const hasValidCustomer = _.has(cleanedOrder, 'customer.name') && !_.isEmpty(cleanedOrder.customer.name);
    
    // Example using uuid for unique identifiers
    const processingId = uuidv4();
    
    return {
        version: '0.1.4',
        _id: inputOrder._id,
        order_number: inputOrder.order_number,
        processed_at: now.toISOString(),
        processed_time: now.format('YYYY-MM-DD HH:mm:ss'),
        processing_id: processingId, // uuid generated
        has_valid_customer: hasValidCustomer, // lodash validation
        field_count: _.keys(cleanedOrder).length, // lodash utility
        // ... other fields
    };
}

​
Key Files You’ll Work With

New Project: 
src/
├── OrderTransformer.ts     # Your main business logic goes here
├── OrderType1.ts          # Define what your input data looks like
├── MergedOrder.ts         # Define what your output data looks like
├── *.test.ts             # Automated tests (optional to modify)
└── index.ts              # System file (don't modify)
Existing Project: 
src/templates/
├── commonTransform.ts     # Connect your existing services here
├── valueTransform.ts      # Main value transformation logic
├── keyTransform.ts        # Key transformation for partitioning
├── topicTransform.ts      # Topic routing for fan-out
└── index.ts              # System file (don't modify)
Both approaches generate: 
transforms/               # Generated files (copy these to Streamkap)

​
What You Get

  • Type Safety: Catch errors before deployment with TypeScript
  • Rich Ecosystem: Use thousands of npm packages in your transforms
  • Testing: Automated tests ensure your code works correctly
  • Self-Contained: Generated files include everything needed - no external dependencies
  • All Transform Types: Works with every Streamkap transform type
  • Two Integration Options: Use as standalone project or integrate into existing codebase
  • Standard Workflow: Familiar git-based development process

​
Troubleshooting

Tests failing? 
npm run build  # Build first - tests validate generated files
npm test -- --verbose
Build errors? 
npm install
npm run build 2>&1 | grep -i error
Only need specific transforms? 
# Build individual transform types for faster development
npm run build:map-filter      # Only builds map/filter transforms
npm run test:enrich-async     # Builds and tests async enrichment

# Test without building (uses existing builds)
npm test -- --map-filter     # Test only map/filter transforms
npm test -- --fan-out --un-nesting  # Test multiple specific types

# Build multiple specific types
node build-multiple.js --fan-out --un-nesting

# Available types: --map-filter, --fan-out, --enrich-async, --un-nesting, --all
Streamkap deployment not working?
  • Make sure you copy the entire file contents (including all the bundled code)
  • Always run npm test locally before deploying
  • Check the Streamkap error console for specific error messages
Always run npm run build before npm test. The tests check the actual files that get deployed to Streamkap.

​
Need More Details?

This guide covers the basics. For advanced usage, troubleshooting, and detailed explanations, check the README.md file in the repository.
⌘I