> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy a Turbo pipeline

> Deploy your first Turbo Pipeline in minutes

## Overview

In this quickstart, you'll create a simple Turbo pipeline that:

1. **Reads ERC-20 transfer data** from a Goldsky dataset
2. **Writes the results to a blackhole sink**
3. **Inspect the data live**
4. **\[Optional] Write the data into a PostgreSQL Database**

You have two options to create a Turbo pipeline:

1. **[Goldsky Flow](#goldsky-flow)**: A guided visual canvas editor in the dashboard
2. **[CLI](#creating-turbo-pipelines-with-the-cli)**: Using YAML configuration files

## Goldsky Flow

Flow allows you to deploy Turbo pipelines by dragging and dropping components onto a visual canvas. Open Flow by going to the [Pipelines page](https://app.goldsky.com/dashboard/pipelines) and clicking **New pipeline**.

<Note>
  Turbo pipelines in Flow work similarly to Mirror pipelines but with some differences. See [Turbo vs Mirror in Flow](#turbo-vs-mirror-in-flow) for details.
</Note>

### 1. Select a data source

Drag a **Data Source** card onto the canvas. Select the chain and dataset you want to use. For blockchain data, Turbo supports:

* **EVM chains**: Ethereum, Base, Polygon, and more
* **Solana**: Transactions, instructions, and token transfers
* **Stellar**: Ledgers, transactions, and operations
* **Bitcoin**: Blocks and transactions

Select your chain and dataset type (e.g., `ERC-20 Transfers` for Base).

### 2. Add transforms (optional)

Click the `+` button on your source card to add transforms:

* **SQL Transform**: Filter and project data using SQL queries
* **Script Transform**: Execute custom TypeScript code for complex transformations
* **Dynamic Table**: Create real-time lookup tables for filtering and enrichment

<Tip>
  Dynamic Table and Script transforms are unique to Turbo pipelines and provide powerful capabilities not available in Mirror.
</Tip>

### 3. Select a sink

Click the `+` button to add a sink. Turbo supports:

* PostgreSQL
* ClickHouse
* Kafka
* Webhook
* S3
* Blackhole (for testing)

### 4. Deploy

Name your pipeline and click **Deploy**. Select a resource size and your pipeline will start processing data.

### Switching between Flow and YAML

You can toggle between the visual canvas and YAML view using the switcher in the top-left corner. This lets you:

* See the YAML configuration generated from your visual design
* Copy the YAML for version control or CI/CD deployment
* Make advanced edits directly in YAML

### Turbo vs Mirror in Flow

When creating Turbo pipelines in Flow, note these differences from Mirror:

| Feature                  | Turbo            | Mirror    |
| ------------------------ | ---------------- | --------- |
| Script transforms        | Yes (TypeScript) | No        |
| Dynamic Table transforms | Yes              | No        |
| Kafka source             | YAML only        | Supported |
| ClickHouse source        | YAML only        | Supported |
| Hybrid source            | YAML only        | Supported |
| Live inspect             | Yes              | No        |

<Warning>
  Some source types (Kafka, ClickHouse, and Hybrid) are only available via YAML configuration. Use the [CLI workflow](#creating-turbo-pipelines-with-the-cli) for these advanced sources.
</Warning>

***

## Creating Turbo pipelines with the CLI

## Prerequisites

* [Turbo Pipelines CLI extension installed](/turbo-pipelines/cli#installation)
* A Goldsky account, logged in to your project

## Step 1: Create your pipeline

Create a file named `erc20-pipeline.yaml`:

```yaml erc20-pipeline.yaml theme={null}
name: erc20-transfers
resource_size: s

sources:
  base_erc20_transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms: {}

sinks:
  blackhole_sink:
    type: blackhole
    from: base_erc20_transfers
```

<Accordion title="What's happening in this pipeline?">
  **Sources:**

  * We're using the `base.erc20_transfers` dataset (version 1.2.0)
  * `start_at: latest` means we'll only process new transfers going forward

  **Sinks:**

  * The `blackhole` sink discards the data but allows you to test the pipeline
  * Perfect for development and testing before adding real outputs
</Accordion>

## Step 2: Deploy Your Pipeline

Apply your pipeline configuration:

```bash theme={null}
goldsky turbo apply erc20-pipeline.yaml
```

You should see output confirming the deployment:

```
✓ Pipeline 'erc20-transfers' applied successfully
```

## Step 3: Inspect the Data Live

<Info>
  Now that the pipeline is running, you can pass the pipeline name
  `erc20-transfers` instead of `erc20-pipeline.yaml` to any of the commands
  below.
</Info>

Open the live inspect TUI to watch data flow through your pipeline:

```bash theme={null}
goldsky turbo inspect erc20-pipeline.yaml
```

Use the logs command to stream the pipeline's runtime output:

```bash theme={null}
goldsky turbo logs erc20-pipeline.yaml
```

You should see output showing the pipeline processing ERC-20 transfers in real-time.

## Step 4: Add a Filter for USDC Only

Now let's add a SQL transform to filter only USDC transfers. Update your `erc20-pipeline.yaml`:

```yaml erc20-pipeline.yaml theme={null}
name: erc20-transfers
resource_size: s

sources:
  base_erc20_transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  # Filter to only USDC transfers
  usdc_transfers:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        sender,
        recipient,
        amount,
        to_timestamp(block_timestamp) as block_time
      FROM base_erc20_transfers
      WHERE address = lower('0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913')

sinks:
  blackhole_sink:
    type: blackhole
    from: usdc_transfers
```

<Accordion title="What changed?">
  **Transforms:**

  * Added a SQL transform named `usdc_transfers` that filters for the USDC contract on Base
  * The contract address `0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913` is USDC on Base
  * We use `lower()` to ensure consistent case-insensitive matching
  * Selected only essential columns: `sender`, `recipient`, `amount`, and `block_time`
  * Converted the Unix timestamp to a human-readable format using `to_timestamp()`

  **Sinks:**

  * Updated the sink to read from `usdc_transfers` instead of the raw source
  * Now only USDC transfers will flow through the pipeline
</Accordion>

<Accordion title="Alternative: Using TypeScript Transform">
  You can achieve the same filtering using a TypeScript transform instead of SQL. Return `null` to filter out records:

  ```yaml theme={null}
  name: erc20-transfers
  resource_size: s

  sources:
    base_erc20_transfers:
      type: dataset
      dataset_name: base.erc20_transfers
      version: 1.2.0
      start_at: latest

  transforms:
    usdc_transfers:
      type: script
      primary_key: id
      language: typescript
      from: base_erc20_transfers
      schema:
        id: string
        sender: string
        recipient: string
        amount: string
        block_time: string
      script: |
        function invoke(data) {
          const USDC_ADDRESS = '0x833589fcd6edb6e08f4c7c32d4f71b54bda02913';

          // Return null to filter out non-USDC transfers
          if (!data?.address || data.address.toLowerCase() !== USDC_ADDRESS) {
            return null;
          }

          // Return a custom schema with only the fields we need
          return {
            id: data.id,
            sender: data.sender,
            recipient: data.recipient,
            amount: data.amount,
            block_time: new Date(data.block_timestamp * 1000).toISOString()
          };
        }

  sinks:
    blackhole_sink:
      type: blackhole
      from: usdc_transfers
  ```

  **Key features:**

  * **Return `null` to filter**: Records that don't match your criteria can be filtered out by returning `null`
  * **Custom output schema**: Use the `schema` field to define a different output schema than the input. This lets you reshape data, rename fields, or include only specific columns
  * **Flexible transformations**: Perform calculations, string manipulation, and conditional logic

  **TypeScript Benefits:**

  * More flexible data transformations and complex logic
  * Familiar syntax for developers
  * Type safety and autocompletion support
  * Can perform calculations, string manipulation, and conditional logic

  **SQL Benefits:**

  * Generally faster for simple filtering and aggregations
  * More concise for straightforward queries
  * Better for set-based operations

  Choose TypeScript when you need complex transformations or custom logic. Choose SQL for simple filters and aggregations.
</Accordion>

### Redeploy and Inspect

Apply the updated configuration:

```bash theme={null}
goldsky turbo apply erc20-pipeline.yaml
```

Now inspect the output of your new `usdc_transfers` transform.

```bash theme={null}
# the -n flag lets you target one or more sources or transforms
goldsky turbo inspect erc20-pipeline.yaml -n usdc_transfers
```

You should now see only USDC transfers! Compare this to the original source data:

```bash theme={null}
# View the filtered USDC data
goldsky turbo inspect erc20-pipeline.yaml -n usdc_transfers

# View all ERC-20 transfers (before filtering)
goldsky turbo inspect erc20-pipeline.yaml -n base_erc20_transfers

# View multiple stages at once
goldsky turbo inspect erc20-pipeline.yaml -n base_erc20_transfers,usdc_transfers
```

<Tip>
  The `-n, --topology-node-keys` parameter lets you inspect data at any point
  in your pipeline. Use transform names to see filtered/transformed data, or
  source names to see raw data. You can specify multiple keys separated by
  commas to view multiple stages simultaneously.
</Tip>

# Optional: Write to PostgreSQL

To persist your data to a PostgreSQL database, update your pipeline configuration:

### 1. Create a Secret

Store your PostgreSQL credentials:

```bash theme={null}
goldsky secret create MY_POSTGRES_SECRET
```

When prompted, enter your PostgreSQL connection string:

```
postgres://username:password@host:port/database
```

### 2. Update Your Pipeline

Modify `erc20-pipeline.yaml` to add a PostgreSQL sink:

```yaml erc20-pipeline.yaml theme={null}
name: erc20-transfers
resource_size: s

sources:
  base_erc20_transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  # Filter to only USDC transfers
  usdc_transfers:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        sender,
        recipient,
        amount,
        to_timestamp(block_timestamp) as block_time
      FROM base_erc20_transfers
      WHERE address = lower('0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913')

sinks:
  postgres_sink:
    type: postgres
    schema: public
    table: usdc_transfers
    secret_name: MY_POSTGRES_SECRET
    from: usdc_transfers
    primary_key: id
```

### 3. Redeploy

Apply the updated configuration:

```bash theme={null}
goldsky turbo apply erc20-pipeline.yaml
```

### 4. Query Your Data

Connect to PostgreSQL and query the USDC transfers:

```sql theme={null}
SELECT
  sender,
  recipient,
  amount,
  block_time
FROM public.usdc_transfers
ORDER BY block_time DESC
LIMIT 10;
```