Indexing Movement with Goldsky

​

Overview

Goldsky is a high-performance data indexing provider for Movement that makes it easy to extract, transform, and load on-chain data to power both application and analytics use cases. Goldsky primarily offers indexing and accessing blockchain data via Mirror (real-time data replication pipelines).

​

Getting started

To use Goldsky, you’ll need to create an account, install the CLI, and log in.

​

Mirror

Mirror pipelines allow users to replicate data into their own infrastructure (any of the supported sinks) in real time. The supported data sources can be found here.

For a complete overview of how to deploy Mirror pipelines, including a video walkthrough, check out Create a Pipeline. Below, we will look at a few of the different ways by which you can deploy Mirror pipelines; Here, we’ll be streaming the movement.raw_events direct indexing dataset into a ClickHouse database.

Pipelines can be deployed on Goldsky in 3 ways:

• Using Goldsky Flow in the dashboard:
  • Drop a Data Source card. Select Movement as the chain. Then Movement Raw Events as the onchain dataset to use.
  • Add a Sink card and select your pre-configured sink.
  • Deploy pipeline.
• Using the interactive CLI:
  • Enter command goldsky pipeline create <pipeline-name>.
  • This will kick off a guided flow with the first step to choose the dataset type. Choose Direct Indexing.
  • Next, select Movement as the chain with the enter key, then Movement Raw Events as the dataset by pressing space on the selected dataset which you can select using the up and down arrow keys. We will process historical data so select Process all historical data by pressing the enter key. This is the same as the yaml config setting of start_at: earliest. If you prefer to only ingest data starting when your pipeline is deployed, select Process data from the time this pipeline is created instead, this is equivalent to the yaml config setting start_at: latest.
  • When asked to choose another source, choose No by pressing enter.
  • Add your pre-configured sink. In this example case, we chose ClickHouse and as a next step we selected the database schema.
  • Pipeline will be deployed automatically.
• Using a pipeline configuration file:
  • This makes it easier to set up complex pipelines involving multiple sources, multiple sinks, and more complex, SQL-based transformations. For the full reference documentation on, click here.
  • As in the previous steps, we’ll be deploying a pipeline to stream Movement Raw Events. Unlike the other methods, we have added a transformation to only select specific data attributes from the whole dataset. This is the configuration file:

name: movement-raw-events
resource_size: s
apiVersion: 3
sources:
  source_1:
    type: dataset
    dataset_name: movement.raw_events
    version: 1.0.0
    start_at: earliest
transforms:
  sql_1:
    type: sql
    sql: SELECT account_address, type, data, inserted_at from source_1
    primary_key: id
sinks:
  sink_1:
    type: clickhouse
    secret_name: <YOUR_SECRET_NAME>
    from: sql_1
    table: sql_1

• Add your corresponding secret name and run goldsky pipeline apply sui-transactions.yaml --status ACTIVE to deploy the pipeline.

​

Getting support

Can't find what you're looking for? Reach out to us at support@goldsky.com for help.