Streaming Stellar data with Goldsky

Overview

Goldsky is a high-performance data indexing provider for Stellar that makes it easy to extract, transform, and load on-chain data to power both application and analytics use cases via Mirror (real-time data replication pipelines).

Getting started

To use Goldsky, you’ll need to create an account, install the CLI, and log in.

Install Goldsky's CLI and log in

Install the Goldsky CLI: For macOS/Linux:
```
curl https://goldsky.com | sh
```
For Windows:
```
npm install -g @goldskycom/cli
```
Windows users need to have Node.js and npm installed first. Download from nodejs.org if not already installed.
Go to your Project Settings page and create an API key.
Back in your Goldsky CLI, log into your Project by running the command goldsky login and paste your API key.
Now that you are logged in, run goldsky to get started:
```
goldsky
```

Mirror

Mirror pipelines allow users to replicate data into their own infrastructure (any of the supported sinks) in real time. For a complete overview of how to deploy Mirror pipelines, including a video walkthrough, check the Create a Pipeline. Below, we will first look into the data model for Stellar primitives and then see some example implementation pipelines that you can use as a reference.

Working with the Stellar Datasets

Goldsky provides real time (<5 seconds) streaming of Stellar datasets, including all historical data, for both mainnet and testnet. Every Stellar dataset is derived from a main Ledgers dataset. Ledgers are the core building blocks of the Stellar blockchain and the highest level of abstraction in our datasets.

Need a refresher on Stellar data structures? Check out Stellar’s official documentation.

As an example, let’s look at Ledger 57255191. A ledger contains multiple transactions, and each transaction contains one or more operations, at the same time, each transaction and operation contains 0:n events. We’ve modeled these inner structures accordingly in the datasets:

transfers
transactions
diagnostic_events
events
balances

These are the datasets that you can see enabled on the Goldsky dashboard and it’s the preferred way of working with this data. Advanced - Working with Ledgers dataset on the CLI

Working with ledgers involves a certain level of SQL knowledge which is why it’s easier to directly work with the inner datasets directly using the dashboard experience whenever it’s possible.

Advanced users that feel comfortable working with the Goldsky CLI have the option to use the canonical ledgers dataset itself. In essence, ledgers functions as a “mega-schema,” allowing you to query all ledger data holistically or customize exactly which subsets of data you want.

Ledger Schema

{
    "type": "record",
    "name": "Ledger",
    "namespace": "com.stellar.flatten",
    "fields": [
        {
            "name": "sequence",
            "type": "long"
        },
        {
            "name": "ledger_hash",
            "type": "string"
        },
        {
            "name": "previous_ledger_hash",
            "type": "string"
        },
        {
            "name": "closed_at",
            "type": {
                "type": "long",
                "logicalType": "timestamp-millis"
            }
        },
        {
            "name": "protocol_version",
            "type": "int"
        },
        {
            "name": "total_coins",
            "type": "long"
        },
        {
            "name": "fee_pool",
            "type": "long"
        },
        {
            "name": "base_fee",
            "type": "int"
        },
        {
            "name": "base_reserve",
            "type": "int"
        },
        {
            "name": "max_tx_set_size",
            "type": "int"
        },
        {
            "name": "successful_transaction_count",
            "type": "int"
        },
        {
            "name": "failed_transaction_count",
            "type": "int"
        },
        {
            "name": "soroban_fee_write_1kb",
            "type": [
                "null",
                "long"
            ],
            "default": null
        },
        {
            "name": "node_id",
            "type": [
                "null",
                "string"
            ],
            "default": null
        },
        {
            "name": "signature",
            "type": [
                "null",
                "string"
            ],
            "default": null
        },
        {
            "name": "transactions",
            "type": {
                "type": "array",
                "items": {
                    "type": "record",
                    "name": "Transaction",
                    "fields": [
                        {
                            "name": "transaction_hash",
                            "type": "string"
                        },
                        {
                            "name": "account",
                            "type": "string"
                        },
                        {
                            "name": "account_muxed",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "account_sequence",
                            "type": "long"
                        },
                        {
                            "name": "max_fee",
                            "type": "long"
                        },
                        {
                            "name": "fee_charged",
                            "type": "long"
                        },
                        {
                            "name": "fee_account",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "fee_account_muxed",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "inner_transaction_hash",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "new_max_fee",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "memo_type",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "memo",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "time_bounds_lower",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "time_bounds_upper",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "successful",
                            "type": "boolean"
                        },
                        {
                            "name": "transaction_result_code",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "operation_count",
                            "type": "int"
                        },
                        {
                            "name": "inclusion_fee_bid",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "resource_fee",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_instructions",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_read_bytes",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_write_bytes",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "non_refundable_resource_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "refundable_resource_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "rent_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "tx_signers",
                            "type": {
                                "type": "array",
                                "items": "string"
                            },
                            "default": []
                        },
                        {
                            "name": "operations",
                            "type": {
                                "type": "array",
                                "items": {
                                    "type": "record",
                                    "name": "Operation",
                                    "fields": [
                                        {
                                            "name": "id",
                                            "type": "string"
                                        },
                                        {
                                            "name": "source_account",
                                            "type": "string"
                                        },
                                        {
                                            "name": "source_account_muxed",
                                            "type": "string"
                                        },
                                        {
                                            "name": "type",
                                            "type": "string"
                                        },
                                        {
                                            "name": "body",
                                            "type": "string"
                                        },
                                        {
                                            "name": "result_code",
                                            "type": "string"
                                        },
                                        {
                                            "name": "ledger_entry_changes",
                                            "type": {
                                                "type": "array",
                                                "items": {
                                                    "type": "record",
                                                    "name": "LedgerEntryChange",
                                                    "fields": [
                                                        {
                                                            "name": "id",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "change_type",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "ledger_entry_type",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "entry_data",
                                                            "type": [
                                                                "null",
                                                                "string"
                                                            ],
                                                            "default": null
                                                        },
                                                        {
                                                            "name": "key_data",
                                                            "type": [
                                                                "null",
                                                                "string"
                                                            ],
                                                            "default": null
                                                        },
                                                        {
                                                            "name": "last_modified_ledger_sequence",
                                                            "type": [
                                                                "null",
                                                                "long"
                                                            ],
                                                            "default": null
                                                        }
                                                    ],
                                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                                }
                                            }
                                        }
                                    ],
                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                }
                            }
                        },
                        {
                            "name": "events",
                            "type": {
                                "type": "array",
                                "items": {
                                    "type": "record",
                                    "name": "Event",
                                    "fields": [
                                        {
                                            "name": "id",
                                            "type": "string"
                                        },
                                        {
                                            "name": "type",
                                            "type": "string"
                                        },
                                        {
                                            "name": "contract_id",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "topics",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "data",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "in_successful_contract_call",
                                            "type": "boolean"
                                        }
                                    ],
                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                }
                            },
                            "default": []
                        }
                    ],
                    "connect.doc": "{\"primaryKeys\":[\"transaction_hash\"]}"
                }
            }
        }
    ],
    "connect.doc": "{\"primaryKeys\":[\"sequence\"]}"
}

You can view the excerpt of an example record of the ledgers dataset here. To use this dataset, choose stellar.ledgers as the dataset_name, specify latest version 3.1.0 and start_at as earliest or latest depending on whether you want historical data or data from the current tip of the chain:

sources:
  ledgers:
    type: dataset
    dataset_name: stellar.ledgers
    version: 3.1.0
    start_at: latest|earliest

From here you can then add any corresponding transformations to access its inner datastructure as explained in the later sections.

1. Deploying Stellar inner datasets using Goldsky Flow

As explained in Create a Pipeline, Goldsky Flow is the visual editor from which we can build and deploy Mirror pipelines. In this example we create a pipeline using Flow to stream into the transactions dataset into a ClickHouse instance. Start from the pipelines page in your project and click on New Pipeline

Screenshot 2025-11-06 at 11.57.56 AM.png

This will redirect you an open canvas where you can build your pipeline by dragging and dropping components. From here, you’ll want to drop a Source card in to the canvas and select Stellar as the network:

Screenshot 2025-11-06 at 11.58.23 AM.png

From the list of specialized datasets choose Transactions

Screenshot 2025-11-06 at 11.58.30 AM.png

Next, we need to add a Sink for the data. You can click on + button on the source card and select a Sink

Screenshot 2025-11-06 at 11.58.38 AM.png

In this case, we already have a few sinks preconfigured for Postgres and ClickHouse. In this example, we’ll go for the sink CLICKHOUSE_DEMO

Screenshot 2025-11-06 at 11.58.54 AM.png

As a good practice, we’ll be renaming the sink table that Mirror will create as transactions

Screenshot 2025-11-06 at 11.59.03 AM.png

Next, we specify a name for the pipeline stellar-transactions

Screenshot 2025-11-06 at 12.00.44 PM.png

We are ready to deploy: click on the Deploy button on the top right of your screen. You can deploy this pipeline with Small resource size.

Screenshot 2025-11-06 at 12.00.50 PM.png

This will redirect us to the pipeline details page. After some seconds the pipeline will be Live and streaming data onto our sink

Screenshot 2025-11-06 at 12.12.30 PM.png

Here we can see a sample of the edge transactions data in our ClickHouse instance

Screenshot 2025-11-06 at 12.18.13 PM.png

2. Advanced: examples using Stellar Ledgers on the CLI

As mentioned earlier, an alternative way of deploying pipelines is using a pipeline definition file and deploying them using the CLI. Working with the ledgers dataset usually involves using a combination of CROSS JOIN UNNEST queries to “explode” and be able to access its inner structures. The example below demonstrates how we can replicate the transactions pipeline from the previous example using Goldsky Flow by exploding the data from the ledgers dataset:

name: stellar-ledger-transactions
resource_size: s
apiVersion: 3
sources:
  ledgers:
    type: dataset
    dataset_name: stellar.ledgers
    version: 3.1.0
    start_at: latest
transforms:
  sql_1:
    type: sql
    sql: |-
      SELECT
              ledgers.sequence AS ledger_sequence,
              transaction.transaction_hash as transaction_hash,
              transaction.account as account,
              transaction.transaction_result_code as transaction_result_code
            FROM
              ledgers
            CROSS JOIN UNNEST(transactions) AS transaction
    primary_key: transaction_hash
sinks:
  sink_1:
    type: clickhouse
    secret_name: <YOUR_SECRET_NAME>
    from: sql_1
    table: stellar_events

As another example: balances are also directly exposed in the dataset balances but here’s the actual way these are computed directly from the master ledgers dataset in case you’d like to create your own modification:

name: stellar-balances
version: 1
resource_size: s
apiVersion: 3
sources:
  ledgers:
    type: dataset
    dataset_name: stellar.ledgers
    version: 3.1.0
    start_at: latest
transforms:
  # Ledger entry changes for payments:
  #   * Change types "update" or "created" (for post-change)
  #   * Entry types "account" (XLM) or "trustline" (token)
  payment_changes:
    type: sql
    primary_key: change_id
    sql: |
      SELECT
        ledgers.sequence AS ledger_sequence,
        change.id AS change_id,
        change.ledger_entry_type,
        change.entry_data
      FROM
        ledgers
      -- explode nested arrays
      CROSS JOIN UNNEST(transactions) AS transaction
      CROSS JOIN UNNEST(transaction.operations) AS operation
      CROSS JOIN UNNEST(operation.ledger_entry_changes) AS change
      WHERE
        -- 1) Only “payment” operations:
        operation.type = 'payment'
        -- 2) Only post‐change records (new balance):
        AND change.change_type IN ('updated', 'created')
        -- 3) Only XLM (“account”) or token (“trustline”) ledger entries:
        AND change.ledger_entry_type IN ('account', 'trustline')
      ;

  # Normalized balances. Take each change row (from payment_changes) and extract:
  #   * account_id
  #   * asset_code  (XLM if ledger_entry_type='account', else trustline code)
  #   * asset_issuer (NULL if XLM, else trustline issuer)
  #   * balance_stroops (new balance, in stroops)
  normalized_balances:
    type: sql
    primary_key: change_id
    sql: |
      SELECT
        pc.ledger_sequence,
        pc.change_id,
        pc.ledger_entry_type,

        -- Extract account_id:
        CASE
          WHEN pc.ledger_entry_type = 'account'
            THEN JSON_VALUE(pc.entry_data, '$.accountID')
          ELSE JSON_VALUE(pc.entry_data, '$.trustline.account_id')
        END AS account_id,

        -- Determine asset_code:
        CASE
          WHEN pc.ledger_entry_type = 'account'
            THEN 'XLM'
          ELSE COALESCE(
            JSON_VALUE(pc.entry_data, '$.trustline.asset.credit_alphanum4.asset_code'),
            JSON_VALUE(pc.entry_data, '$.trustline.asset.credit_alphanum12.asset_code')
          )
        END AS asset_code,

        -- Determine asset_issuer:
        CASE
          WHEN pc.ledger_entry_type = 'account'
            THEN CAST(NULL AS STRING)
          ELSE COALESCE(
            JSON_VALUE(pc.entry_data, '$.trustline.asset.credit_alphanum4.issuer'),
            JSON_VALUE(pc.entry_data, '$.trustline.asset.credit_alphanum12.issuer')
          )
        END AS asset_issuer,

        -- New balance in stroops:
        CAST(
          CASE
            WHEN pc.ledger_entry_type = 'account'
              THEN JSON_VALUE(pc.entry_data, '$.balance')
            ELSE JSON_VALUE(pc.entry_data, '$.trustline.balance')
          END AS BIGINT
        ) AS balance_stroops

      FROM payment_changes AS pc
      ;
  # Build the final balances table with a new primary key
  balances_final:
    type: sql
    primary_key: balance_key
    sql: |
      SELECT
        nb.ledger_sequence,
        nb.account_id,
        nb.asset_code,
        nb.asset_issuer,
        nb.balance_stroops,

        -- Build composite upsert key. If asset_issuer is NULL, COALESCE→''.
        CONCAT(
          COALESCE(nb.account_id, ''),
          '-',
          COALESCE(nb.asset_code, ''),
          '-',
          COALESCE(nb.asset_issuer, '')
        ) AS balance_key

      FROM normalized_balances AS nb
      WHERE
        -- Ensure account_id is present:
        nb.account_id IS NOT NULL
        -- Ensure asset_code is present:
        AND nb.asset_code IS NOT NULL
      ;

sinks:
  sink_1:
    type: clickhouse
    table: stellar_balances
    secret_name: <YOUR_SECRET_NAME>
    from: balances_final

use_dedicated_ip: false

You can deploy these pipeline using the CLI by first configuring your sink and running goldsky pipeline apply <file_name> --status ACTIVE

Working with Testnet data

When working with Stellar testnet data, it’s important to note that the testnet is frequently reset—typically about once every third month. Each reset effectively starts a new version of the testnet, but the data continues to flow into the same datasets (e.g., ledgers, transactions, etc.). While you don’t need to take any specific action when a reset occurs, we recommend deploying your pipelines with start_at: latest. This ensures that your pipelines always index from the most recent testnet version, preventing data from older and newer testnet instances from becoming intermingled.

Getting support

Can’t find what you’re looking for? Reach out to us at support@goldsky.com for help.

​Overview

​Getting started

​Mirror

​Working with the Stellar Datasets

​1. Deploying Stellar inner datasets using Goldsky Flow

​2. Advanced: examples using Stellar Ledgers on the CLI

​Working with Testnet data

​Getting support

Overview

Getting started

Mirror

Working with the Stellar Datasets

1. Deploying Stellar inner datasets using Goldsky Flow

2. Advanced: examples using Stellar Ledgers on the CLI

Working with Testnet data

Getting support