Streaming Sui data with Goldsky
Overview
Goldsky is a high-performance data indexing provider for Sui that makes it easy to extract, transform, and load on-chain data to power both application and analytics use cases via Mirror (real-time data replication pipelines).
Scope
Goldsky has partnered with Sui to make our product available to the ecosystem and provide dedicated support for Sui data indexing. The full scope of our partnership (which products are enabled, what partner-exclusive benefit is available, and who this benefit is available to) is outlined below.
Mirror | |
---|---|
Enablement | Yes |
Benefit | 10% discount on Pipeline workers and events written |
Availability | All developers |
Where perks are available to projects specified by the chain ecosystem, please reach out to their developer relations team for an access code and link to private signup form. Where perks are openly available to all developers, please contact sales@goldsky.com to apply the applicable partnership perks to your project.
Getting started
To use Goldsky, you’ll need to create an account, install the CLI, and log in.
Mirror
Mirror pipelines allow users to replicate data into their own infrastructure (any of the supported sinks) in real time. The supported data sources are the following chain-level datasets: checkpoints, packages, transactions, epochs and events.
For a complete overview of how to deploy Mirror pipelines, including a video walkthrough, check the Create a Pipeline. Below, we will look at a high level into the different ways by which you can deploy Mirror pipelines; in specific, we’ll be streaming the sui.transactions
chain-level dataset into a PostgreSQL database.
Remember to first create a Secret in order for Mirror to be able to write the data into the database of your choice.
Pipelines can be deployed on Goldsky in 3 ways:
- Using the Pipeline Builder in the Goldsky dashboard:
- Select
Direct Indexing
as data source filter on Sui and selectTransactions
as dataset. - Follow the default options in the next steps and choose your pre-configured sink.
- Deploy pipeline.
- Select
- Using the interactive CLI:
- Enter command
goldsky pipeline create <pipeline-name>
. - This will kick off a guided flow with the first step to choose the dataset type. Choose
chain-level dataset
. - Next, select
Sui
as chain andRaw Transactions
as the dataset. We will process historical data so selectProcess all historical data
. - When asked to choose another source, choose
No
- Add your pre-configured sink. In this example case, we chose PostgreSQL and as a next step we selected the database schema.
- Pipeline will be deployed automatically.
- Enter command
- Using a pipeline configuration file:
- This makes it easier to set up complex pipelines involving multiple sources, multiple sinks, and more complex, SQL-based transformations. For the full reference documentation on, click here.
- As in the previous steps, we’ll be deploying a pipeline to stream Sui transactions. Unlike the other methods, we have added a transformation to only select specific data attributes from the whole dataset. This is the configuration file:
apiVersion: 3
name: sui-raw-transactions
status: ACTIVE
sources:
sui_transactions:
dataset_name: sui.transactions
version: 1.0.0
type: dataset
start_at: earliest
transforms:
select_relevant_fields:
primary_key: tx_sequence_number
sql: |
SELECT
tx_sequence_number,
checkpoint_sequence_number,
effects,
sender_signed_data
FROM
sui_transactions
sinks:
postgres_sui_transactions:
type: postgres
table: sui_transactions
schema: public
secret_name: <YOUR_SECRET_NAME>
description: 'Postgres sink for: sui_transactions'
from: sui_transactions
- Add your corresponding secret name and run
goldsky pipeline apply sui-transactions.yaml
to deploy the pipeline.
Getting support
Can't find what you're looking for? Reach out to us at support@goldsky.com for help.