Goldsky Documentation

Overview

The TypeScript transform allows you to execute custom TypeScript code on each record in your pipeline. This is useful for:

Complex data transformations not supported by SQL
Custom business logic with type safety
Data parsing and formatting
Conditional transformations based on complex rules

Code runs in a sandboxed WebAssembly environment for security and performance.

Typescript transforms may be a lot less efficient compared to SQL, so use SQL when possible first to filter before passing into a typescript transform!

Configuration

transforms:
  my_script:
    type: script
    from: <source-or-transform>
    language: typescript
    primary_key: <column-name>
    script: |
      function invoke(data) {
        // Your TypeScript code here
        return data;
      }

Parameters

type

string

required

Must be script

from

string

required

The source or transform to read data from

language

string

required

Must be typescript

primary_key

string

required

The column that uniquely identifies each row

schema

object

Define a custom output schema when you want to return different fields than the input. Map field names to types (string, float64, int64, boolean, etc.). If omitted, the output schema matches the input.

script

string

required

Your TypeScript code as an invoke(data) function that receives a record and returns the transformed record. Return null to filter out records.

Script Structure

Your script must define an invoke function that:

Accepts a single data parameter (the record object)
Returns the transformed record object, or null to filter out the record
Can return a different schema than the input when using the schema configuration

Basic Example

transforms:
  add_timestamp:
    type: script
    from: source
    language: typescript
    primary_key: id
    script: |
      function invoke(data) {
        data.processed_at = Date.now();
        data.processed = true;
        return data;
      }

Filtering records

Return null to filter out records that don’t match your criteria:

transforms:
  high_value_only:
    type: script
    from: token_balances
    language: typescript
    primary_key: id
    schema:
      id: string
      amount: float64
    script: |
      function invoke(data) {
        // Filter out records with amount <= 1
        if (data.amount <= 1) {
          return null;
        }
        return { id: data.id, amount: data.amount };
      }

Custom output schema

Use the schema field to define a different output schema than the input. This lets you reshape data, select specific fields, or rename columns:

transforms:
  reshape_data:
    type: script
    from: transfers
    language: typescript
    primary_key: transfer_id
    schema:
      transfer_id: string
      sender: string
      receiver: string
      value_eth: float64
      timestamp: string
    script: |
      function invoke(data) {
        return {
          transfer_id: data.id,
          sender: data.from_address,
          receiver: data.to_address,
          value_eth: Number(data.value) / 1e18,
          timestamp: new Date(data.block_timestamp * 1000).toISOString()
        };
      }

Input/Output Format

The data parameter is a JavaScript object with your record’s fields:

// Input object example
{
  "id": "abc123",
  "address": "0x742d35cc6634c0532925a3b844bc9e7595f0beb",
  "value": "1000000000000000000",
  "block_number": 12345678
}

Return a modified object:

{
  "id": "abc123",
  "address": "0x742d35cc6634c0532925a3b844bc9e7595f0beb",
  "value": "1000000000000000000",
  "block_number": 12345678,
  "value_eth": "1.0",  // Added field
  "is_large": true      // Added field
}

Examples

Example: Type-Safe Value Formatting

Convert wei to ETH with TypeScript type safety:

transforms:
  format_values:
    type: script
    from: ethereum_transfers
    language: typescript
    primary_key: id
    script: |
      interface Transfer {
        id: string;
        from_address: string;
        to_address: string;
        value: string;
        block_number: number;
      }

      type SizeLabel = "whale" | "large" | "normal";

      function invoke(data: Transfer): Transfer & {
        value_eth: string;
        size_label: SizeLabel;
      } {
        // Convert wei to ETH
        const valueWei = BigInt(data.value);
        const valueEth = Number(valueWei) / 1e18;

        // Determine size label
        let size_label: SizeLabel;
        if (valueEth > 1000) {
          size_label = "whale";
        } else if (valueEth > 10) {
          size_label = "large";
        } else {
          size_label = "normal";
        }

        return {
          ...data,
          value_eth: valueEth.toFixed(6),
          size_label
        };
      }

Example: Parse JSON Fields

Extract data from JSON strings with type safety:

transforms:
  parse_metadata:
    type: script
    from: nft_transfers
    language: typescript
    primary_key: id
    script: |
      interface NFTMetadata {
        name?: string;
        description?: string;
        image?: string;
        attributes?: Array<{ trait_type: string; value: string }>;
      }

      interface NFTTransfer {
        id: string;
        token_id: string;
        metadata?: string;
      }

      function invoke(data: NFTTransfer): NFTTransfer & {
        nft_name: string;
        nft_description: string;
        image_url: string;
        attributes_count: number;
        parse_error?: boolean;
      } {
        let nft_name = "Unknown";
        let nft_description = "";
        let image_url = "";
        let attributes_count = 0;
        let parse_error: boolean | undefined;

        if (data.metadata) {
          try {
            const meta: NFTMetadata = JSON.parse(data.metadata);
            nft_name = meta.name || "Unknown";
            nft_description = meta.description || "";
            image_url = meta.image || "";
            attributes_count = meta.attributes?.length || 0;
          } catch (e) {
            parse_error = true;
          }
        }

        return {
          ...data,
          nft_name,
          nft_description,
          image_url,
          attributes_count,
          ...(parse_error && { parse_error })
        };
      }

Example: Complex Conditional Logic

Apply different transformations based on conditions:

transforms:
  categorize_transfers:
    type: script
    from: transfers
    language: typescript
    primary_key: id
    script: |
      interface Transfer {
        id: string;
        from_address: string;
        to_address: string;
        value: string;
      }

      type TransferCategory = "exchange_withdrawal" | "exchange_deposit" | "whale_transfer" | "normal_transfer";

      function invoke(data: Transfer): Transfer & {
        category: TransferCategory;
        exchange_from?: boolean;
        exchange_to?: boolean;
        risk_score: number;
      } {
        const value = BigInt(data.value);
        const from = data.from_address.toLowerCase();
        const to = data.to_address.toLowerCase();

        // Known exchange addresses
        const exchanges: string[] = [
          "0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be",
          "0xd551234ae421e3bcba99a0da6d736074f22192ff"
        ];

        let category: TransferCategory;
        let exchange_from: boolean | undefined;
        let exchange_to: boolean | undefined;
        let risk_score = 0;

        // Categorize transfer
        if (exchanges.includes(from)) {
          category = "exchange_withdrawal";
          exchange_from = true;
          risk_score += 0.3;
        } else if (exchanges.includes(to)) {
          category = "exchange_deposit";
          exchange_to = true;
          risk_score += 0.3;
        } else if (value > BigInt("1000000000000000000000")) {
          category = "whale_transfer";
          risk_score += 0.5;
        } else {
          category = "normal_transfer";
        }

        return {
          ...data,
          category,
          ...(exchange_from && { exchange_from }),
          ...(exchange_to && { exchange_to }),
          risk_score
        };
      }

Example: String Manipulation

Clean and format text data:

transforms:
  clean_data:
    type: script
    from: source
    language: typescript
    primary_key: id
    script: |
      interface TokenData {
        id: string;
        address?: string;
        from_address?: string;
        to_address?: string;
        symbol?: string;
      }

      function invoke(data: TokenData): TokenData & {
        short_address?: string;
      } {
        // Normalize addresses to lowercase
        if (data.address) {
          data.address = data.address.toLowerCase();
        }
        if (data.from_address) {
          data.from_address = data.from_address.toLowerCase();
        }
        if (data.to_address) {
          data.to_address = data.to_address.toLowerCase();
        }

        // Trim and clean strings
        if (data.symbol) {
          data.symbol = data.symbol.trim().toUpperCase();
        }

        // Extract short address for display
        const short_address = data.address
          ? data.address.substring(0, 10) + "..."
          : undefined;

        return {
          ...data,
          ...(short_address && { short_address })
        };
      }

Example: Array and Object Manipulation

Work with complex data structures:

transforms:
  process_array_data:
    type: script
    from: solana_blocks
    language: typescript
    primary_key: slot
    script: |
      interface Transaction {
        meta?: {
          err: any;
        };
      }

      interface SolanaBlock {
        slot: number;
        transactions?: Transaction[];
      }

      function invoke(data: SolanaBlock): SolanaBlock & {
        transaction_count: number;
        successful_txs: number;
        success_rate: string;
      } {
        let transaction_count = 0;
        let successful_txs = 0;
        let success_rate = "0.00";

        if (data.transactions && Array.isArray(data.transactions)) {
          transaction_count = data.transactions.length;

          successful_txs = data.transactions.filter(
            tx => tx.meta && tx.meta.err === null
          ).length;

          success_rate = transaction_count > 0
            ? (successful_txs / transaction_count * 100).toFixed(2)
            : "0.00";
        }

        return {
          ...data,
          transaction_count,
          successful_txs,
          success_rate
        };
      }

TypeScript Features

Type Safety Benefits

TypeScript provides:

Compile-time type checking: Catch errors before deployment
IntelliSense: Better IDE autocomplete and suggestions
Refactoring support: Safer code changes
Self-documenting code: Types serve as inline documentation

Supported TypeScript Features

Type Annotations

Interface definitions
Type aliases
Union and intersection types
Generic types
Optional properties (?)
Readonly properties

Modern JavaScript

All ES6+ features (arrow functions, destructuring, spread operator)
JSON.parse() and JSON.stringify()
Math object (Math.floor, Math.random, etc.)
Date object
String methods (split, substring, replace, etc.)
Array methods (map, filter, reduce, etc.)
Object methods (Object.keys, Object.values, etc.)
BigInt for large number handling

Type Guards

typeof checks
instanceof checks
Custom type predicates
Discriminated unions

NOT Available

The following features are not available:

require() or import statements (no external modules)
File system access
Network requests (fetch, XMLHttpRequest)
Node.js APIs (process, fs, http, etc.)
Timers (setTimeout, setInterval)
Async/await (code must be synchronous)

Keep your scripts self-contained and use only browser-compatible TypeScript/JavaScript.

Error Handling

Always include error handling in your scripts:

transforms:
  safe_transform:
    type: script
    from: source
    language: typescript
    primary_key: id
    script: |
      interface Record {
        id: string;
        value: string;
        metadata?: string;
      }

      function invoke(data: Record): Record & {
        value_eth?: string;
        parsed_metadata?: any;
        processing_error?: boolean;
        error_message?: string;
      } {
        try {
          const value = BigInt(data.value);
          const value_eth = (Number(value) / 1e18).toFixed(6);

          let parsed_metadata: any;
          if (data.metadata) {
            parsed_metadata = JSON.parse(data.metadata);
          }

          return {
            ...data,
            value_eth,
            ...(parsed_metadata && { parsed_metadata })
          };
        } catch (error) {
          return {
            ...data,
            processing_error: true,
            error_message: error instanceof Error ? error.message : "Unknown error"
          };
        }
      }

If your script throws an unhandled error, the pipeline will retry processing that record. Use try/catch to handle errors gracefully and flag problematic records for later review.

Performance Considerations

Execution Speed

TypeScript is transpiled to JavaScript at runtime (one-time cost per deployment)
Execution is fast but slower than native SQL transforms
Each record is processed individually
Keep scripts simple and avoid expensive operations

Memory Usage

Scripts run in a sandboxed environment with limited memory
Avoid creating large data structures
Process records one at a time, don’t accumulate state
Clean up temporary variables

Optimization Tips

Pre-define types and interfaces outside the function
Use built-in methods (Array.map, filter) instead of manual loops
Avoid nested loops and recursive functions
Cache frequently accessed values in variables

Debugging

Add Debug Fields

Since console.log is not available, add debug fields to your output:

function invoke(data: Record): Record & { debug_original_value: string; debug_new_value: string } {
  const debug_original_value = data.value;

  // Do transformation
  const value = (BigInt(data.value) / BigInt(1e18)).toString();

  return {
    ...data,
    value,
    debug_original_value,
    debug_new_value: value
  };
}

Then query your sink to see the debug fields.

Test Locally

Before deploying, test your logic in a TypeScript playground or Node.js:

interface TestInput {
  id: string;
  value: string;
}

function invoke(data: TestInput): TestInput & { value_eth: string } {
  return {
    ...data,
    value_eth: (Number(BigInt(data.value)) / 1e18).toFixed(6)
  };
}

// Test with sample data
const testData: TestInput = {
  id: "test",
  value: "1000000000000000000"
};

console.log(invoke(testData));
// Output: { id: "test", value: "1000000000000000000", value_eth: "1.000000" }

Best Practices

Use SQL when possible

SQL transforms are faster and more efficient. Only use TypeScript for logic that SQL cannot express.

Define clear types

Define interfaces for your input and output types:

interface Input {
  id: string;
  value: string;
}

function invoke(data: Input): Input & { value_eth: string } {
  // TypeScript will enforce types
  return {
    ...data,
    value_eth: (Number(BigInt(data.value)) / 1e18).toFixed(6)
  };
}

Use null to filter records

Return null to filter out records that don’t match your criteria:

function invoke(data: Record): Record | null {
  if (data.value <= 0) {
    return null;  // Filter out this record
  }
  return data;
}

Handle null and undefined

Always check for null/undefined values:

function invoke(data: Record): Record & { value_eth?: string } {
  if (data.value != null) {
    const value_eth = (Number(BigInt(data.value)) / 1e18).toFixed(6);
    return { ...data, value_eth };
  }
  return data;
}

Use type guards

Validate data types at runtime:

function invoke(data: any): any {
  if (typeof data.value === 'string' && data.value.length > 0) {
    data.value_eth = (Number(BigInt(data.value)) / 1e18).toFixed(6);
  }
  return data;
}

When to Use TypeScript vs SQL vs HTTP Handler

Use Case	Best Transform
Filtering, projections, simple math	SQL - Fastest and most efficient
External API calls, enrichment	HTTP Handler - Access external data
Complex parsing, custom logic	TypeScript - Full programming flexibility with type safety
String manipulation within bounds of SQL functions	SQL - More efficient
Conditional logic based on multiple fields	TypeScript if complex, SQL if simple CASE works
JSON parsing and manipulation	TypeScript - Use JSON.parse() with type safety
Working with BigInt calculations	TypeScript - Native BigInt support
Type-safe data transformations	TypeScript - Compile-time type checking

Next Steps

SQL Transforms

Use SQL for simpler transformations

HTTP Handler

Call external APIs for enrichment

Sinks

Write transformed data to destinations

Transform Overview

Learn about other transform types

​Overview

​Configuration

​Parameters

​Script Structure

​Basic Example

​Filtering records

​Custom output schema

​Input/Output Format

​Examples

​Example: Type-Safe Value Formatting

​Example: Parse JSON Fields

​Example: Complex Conditional Logic

​Example: String Manipulation

​Example: Array and Object Manipulation

​TypeScript Features

​Type Safety Benefits

​Supported TypeScript Features

​NOT Available

​Error Handling

​Performance Considerations

​Debugging

​Add Debug Fields

​Test Locally

​Best Practices

​When to Use TypeScript vs SQL vs HTTP Handler

​Next Steps

SQL Transforms

HTTP Handler

Sinks

Transform Overview

Overview

Configuration

Parameters

Script Structure

Basic Example

Filtering records

Custom output schema

Input/Output Format

Examples

Example: Type-Safe Value Formatting

Example: Parse JSON Fields

Example: Complex Conditional Logic

Example: String Manipulation

Example: Array and Object Manipulation

TypeScript Features

Type Safety Benefits

Supported TypeScript Features

NOT Available

Error Handling

Performance Considerations

Debugging

Add Debug Fields

Test Locally

Best Practices

When to Use TypeScript vs SQL vs HTTP Handler

Next Steps