Streaming Stellar data with Goldsky

Overview

Goldsky is a high-performance data indexing provider for Stellar that makes it easy to extract, transform, and load on-chain data to power both application and analytics use cases via Mirror (real-time data replication pipelines).

Getting started

To use Goldsky, you’ll need to create an account, install the CLI, and log in.

Install Goldsky's CLI and log in

Install the Goldsky CLI: For macOS/Linux:
```
curl https://goldsky.com | sh
```
For Windows:
```
npm install -g @goldskycom/cli
```
Windows users need to have Node.js and npm installed first. Download from nodejs.org if not already installed.
Go to your Project Settings page and create an API key.
Back in your Goldsky CLI, log into your Project by running the command goldsky login and paste your API key.
Now that you are logged in, run goldsky to get started:
```
goldsky
```

Mirror

Mirror pipelines allow users to replicate data into their own infrastructure (any of the supported sinks) in real time. For a complete overview of how to deploy Mirror pipelines, including a video walkthrough, check the Create a Pipeline. Below, we will first look into the data model for Stellar primitives and then see some example implementation pipelines that you can use as a reference.

Working with the Stellar Datasets

Goldsky provides real time (<5 seconds) streaming of Stellar datasets, including all historical data, for both mainnet and testnet. Every Stellar dataset is derived from a main Ledgers dataset. Ledgers are the core building blocks of the Stellar blockchain and the highest level of abstraction in our datasets.

Need a refresher on Stellar data structures? Check out Stellar’s official documentation.

As an example, let’s look at Ledger 57255191. A ledger contains multiple transactions, and each transaction contains one or more operations, at the same time, each transaction and operation contains 0:n events. We’ve modeled these inner structures accordingly in the datasets:

transfers
transactions
diagnostic_events
events

These are the datasets that you can see enabled on the Goldsky dashboard and it’s the preferred way of working with this data. Advanced - Working with Ledgers dataset on the CLI

Working with ledgers involves a certain level of SQL knowledge which is why it’s easier to directly work with the inner datasets directly using the dashboard experience whenever it’s possible.

Advanced users that feel comfortable working with the Goldsky CLI have the option to use the canonical ledgers dataset itself. In essence, ledgers functions as a “mega-schema,” allowing you to query all ledger data holistically or customize exactly which subsets of data you want.

Ledger Schema

{
    "type": "record",
    "name": "Ledger",
    "namespace": "com.stellar.flatten",
    "fields": [
        {
            "name": "sequence",
            "type": "long"
        },
        {
            "name": "ledger_hash",
            "type": "string"
        },
        {
            "name": "previous_ledger_hash",
            "type": "string"
        },
        {
            "name": "closed_at",
            "type": {
                "type": "long",
                "logicalType": "timestamp-millis"
            }
        },
        {
            "name": "protocol_version",
            "type": "int"
        },
        {
            "name": "total_coins",
            "type": "long"
        },
        {
            "name": "fee_pool",
            "type": "long"
        },
        {
            "name": "base_fee",
            "type": "int"
        },
        {
            "name": "base_reserve",
            "type": "int"
        },
        {
            "name": "max_tx_set_size",
            "type": "int"
        },
        {
            "name": "successful_transaction_count",
            "type": "int"
        },
        {
            "name": "failed_transaction_count",
            "type": "int"
        },
        {
            "name": "soroban_fee_write_1kb",
            "type": [
                "null",
                "long"
            ],
            "default": null
        },
        {
            "name": "node_id",
            "type": [
                "null",
                "string"
            ],
            "default": null
        },
        {
            "name": "signature",
            "type": [
                "null",
                "string"
            ],
            "default": null
        },
        {
            "name": "transactions",
            "type": {
                "type": "array",
                "items": {
                    "type": "record",
                    "name": "Transaction",
                    "fields": [
                        {
                            "name": "transaction_hash",
                            "type": "string"
                        },
                        {
                            "name": "account",
                            "type": "string"
                        },
                        {
                            "name": "account_muxed",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "account_sequence",
                            "type": "long"
                        },
                        {
                            "name": "max_fee",
                            "type": "long"
                        },
                        {
                            "name": "fee_charged",
                            "type": "long"
                        },
                        {
                            "name": "fee_account",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "fee_account_muxed",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "inner_transaction_hash",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "new_max_fee",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "memo_type",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "memo",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "time_bounds_lower",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "time_bounds_upper",
                            "type": [
                                "null",
                                "long"
                            ],
                            "default": null
                        },
                        {
                            "name": "successful",
                            "type": "boolean"
                        },
                        {
                            "name": "transaction_result_code",
                            "type": [
                                "null",
                                "string"
                            ],
                            "default": null
                        },
                        {
                            "name": "operation_count",
                            "type": "int"
                        },
                        {
                            "name": "inclusion_fee_bid",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "resource_fee",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_instructions",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_read_bytes",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "soroban_resources_write_bytes",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "non_refundable_resource_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "refundable_resource_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "rent_fee_charged",
                            "type": "long",
                            "default": 0
                        },
                        {
                            "name": "tx_signers",
                            "type": {
                                "type": "array",
                                "items": "string"
                            },
                            "default": []
                        },
                        {
                            "name": "operations",
                            "type": {
                                "type": "array",
                                "items": {
                                    "type": "record",
                                    "name": "Operation",
                                    "fields": [
                                        {
                                            "name": "id",
                                            "type": "string"
                                        },
                                        {
                                            "name": "source_account",
                                            "type": "string"
                                        },
                                        {
                                            "name": "source_account_muxed",
                                            "type": "string"
                                        },
                                        {
                                            "name": "type",
                                            "type": "string"
                                        },
                                        {
                                            "name": "body",
                                            "type": "string"
                                        },
                                        {
                                            "name": "result_code",
                                            "type": "string"
                                        },
                                        {
                                            "name": "ledger_entry_changes",
                                            "type": {
                                                "type": "array",
                                                "items": {
                                                    "type": "record",
                                                    "name": "LedgerEntryChange",
                                                    "fields": [
                                                        {
                                                            "name": "id",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "change_type",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "ledger_entry_type",
                                                            "type": "string"
                                                        },
                                                        {
                                                            "name": "entry_data",
                                                            "type": [
                                                                "null",
                                                                "string"
                                                            ],
                                                            "default": null
                                                        },
                                                        {
                                                            "name": "key_data",
                                                            "type": [
                                                                "null",
                                                                "string"
                                                            ],
                                                            "default": null
                                                        },
                                                        {
                                                            "name": "last_modified_ledger_sequence",
                                                            "type": [
                                                                "null",
                                                                "long"
                                                            ],
                                                            "default": null
                                                        }
                                                    ],
                                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                                }
                                            }
                                        }
                                    ],
                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                }
                            }
                        },
                        {
                            "name": "events",
                            "type": {
                                "type": "array",
                                "items": {
                                    "type": "record",
                                    "name": "Event",
                                    "fields": [
                                        {
                                            "name": "id",
                                            "type": "string"
                                        },
                                        {
                                            "name": "type",
                                            "type": "string"
                                        },
                                        {
                                            "name": "contract_id",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "topics",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "data",
                                            "type": [
                                                "null",
                                                "string"
                                            ],
                                            "default": null
                                        },
                                        {
                                            "name": "in_successful_contract_call",
                                            "type": "boolean"
                                        }
                                    ],
                                    "connect.doc": "{\"primaryKeys\":[\"id\"]}"
                                }
                            },
                            "default": []
                        }
                    ],
                    "connect.doc": "{\"primaryKeys\":[\"transaction_hash\"]}"
                }
            }
        }
    ],
    "connect.doc": "{\"primaryKeys\":[\"sequence\"]}"
}

You can view the excerpt of an example record of the ledgers dataset here. To use this dataset, choose stellar.ledgers as the dataset_name, specify latest version 3.1.0 and start_at as earliest or latest depending on whether you want historical data or data from the current tip of the chain:

sources:
  ledgers:
    type: dataset
    dataset_name: stellar.ledgers
    version: 3.1.0
    start_at: latest|earliest

From here you can then add any corresponding transformations to access its inner datastructure as explained in the later sections.

1. Deploying Stellar inner datasets using Goldsky Flow

As explained in Create a Pipeline, Goldsky Flow is the visual editor from which we can build and deploy Mirror pipelines. In this example we create a pipeline using Flow to stream into the transactions dataset into a ClickHouse instance. Start from the pipelines page in your project and click on New Pipeline

Screenshot 2025-11-06 at 11.57.56 AM.png

This will redirect you an open canvas where you can build your pipeline by dragging and dropping components. From here, you’ll want to drop a Source card in to the canvas and select Stellar as the network:

Screenshot 2025-11-06 at 11.58.23 AM.png

From the list of specialized datasets choose Transactions

Screenshot 2025-11-06 at 11.58.30 AM.png

Next, we need to add a Sink for the data. You can click on + button on the source card and select a Sink

Screenshot 2025-11-06 at 11.58.38 AM.png

In this case, we already have a few sinks preconfigured for Postgres and ClickHouse. In this example, we’ll go for the sink CLICKHOUSE_DEMO

Screenshot 2025-11-06 at 11.58.54 AM.png

As a good practice, we’ll be renaming the sink table that Mirror will create as transactions

Screenshot 2025-11-06 at 11.59.03 AM.png

Next, we specify a name for the pipeline stellar-transactions

Screenshot 2025-11-06 at 12.00.44 PM.png

We are ready to deploy: click on the Deploy button on the top right of your screen. You can deploy this pipeline with Small resource size.

Screenshot 2025-11-06 at 12.00.50 PM.png

This will redirect us to the pipeline details page. After some seconds the pipeline will be Live and streaming data onto our sink

Screenshot 2025-11-06 at 12.12.30 PM.png

Here we can see a sample of the edge transactions data in our ClickHouse instance

Screenshot 2025-11-06 at 12.18.13 PM.png

2. Simple filters on Stellar datasets

You can use transform blocks or the “Advanced” filter functionality to apply filters on these datasets. By cilcking the “view schema” button you can see a preview of all of the various columns which you can filter on. For example for the transfers dataset, some common filters include:

Pairs of asset_issuer + asset_code for a specific token
A list of addresses for the sender / recipient to track specific wallets
contract_id for specific contracts of interest

These filters can be written simply within the Advanced filter text:

asset_code = 'USDC' AND asset_issuer = 'GA5ZSEJYB37JRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN'

Or as full SQL in a dedicated transform block:

SELECT *
FROM source_1
WHERE
	asset_code = 'USDC'
	AND asset_issuer = 'GA5ZSEJYB37JRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN'

There is no differene in performance or outcome between the two approaches; only developer ergonomics; if dealing with a simple transformation the single input “Advanced” filter field is usually sufficient; for complex or multi-line transforms, using a dedicated Transform block is recommended. Other common fields to filter on against each dataset are listed out below:

Operations
- type
- source_account
Transactions
- memo
Events and Diagnostic Events
- contract_id
- topics

3. Advanced: examples using Stellar Ledgers on the CLI

As mentioned earlier, an alternative way of deploying pipelines is using a pipeline definition file and deploying them using the CLI. Working with the ledgers dataset usually involves using a combination of CROSS JOIN UNNEST queries to “explode” and be able to access its inner structures. The example below demonstrates how we can replicate the transactions pipeline from the previous example using Goldsky Flow by exploding the data from the ledgers dataset:

name: stellar-ledger-transactions
resource_size: s
apiVersion: 3
sources:
  ledgers:
    type: dataset
    dataset_name: stellar.ledgers
    version: 3.1.0
    start_at: latest
transforms:
  sql_1:
    type: sql
    sql: |-
      SELECT
              ledgers.sequence AS ledger_sequence,
              transaction.transaction_hash as transaction_hash,
              transaction.account as account,
              transaction.transaction_result_code as transaction_result_code
            FROM
              ledgers
            CROSS JOIN UNNEST(transactions) AS transaction
    primary_key: transaction_hash
sinks:
  sink_1:
    type: clickhouse
    secret_name: <YOUR_SECRET_NAME>
    from: sql_1
    table: stellar_events

You can deploy these pipeline using the CLI by first configuring your sink and running goldsky pipeline apply <file_name> --status ACTIVE

Working with Testnet data

When working with Stellar testnet data, it’s important to note that the testnet is frequently reset—typically about once every third month. Each reset effectively starts a new version of the testnet, but the data continues to flow into the same datasets (e.g., ledgers, transactions, etc.). While you don’t need to take any specific action when a reset occurs, we recommend deploying your pipelines with start_at: latest. This ensures that your pipelines always index from the most recent testnet version, preventing data from older and newer testnet instances from becoming intermingled.

Getting support

Can’t find what you’re looking for? Reach out to us at support@goldsky.com for help.

​Overview

​Getting started

​Mirror

​Working with the Stellar Datasets

​1. Deploying Stellar inner datasets using Goldsky Flow

​2. Simple filters on Stellar datasets

​3. Advanced: examples using Stellar Ledgers on the CLI

​Working with Testnet data

​Getting support

Overview

Getting started

Mirror

Working with the Stellar Datasets

1. Deploying Stellar inner datasets using Goldsky Flow

2. Simple filters on Stellar datasets

3. Advanced: examples using Stellar Ledgers on the CLI

Working with Testnet data

Getting support