Step by step instructions on how to create a Goldsky Mirror pipeline.
New pipeline
button.
Data Source
card onto the canvas. Once you do that, you’ll to need select the chain you are interested in. We currently support 100+ chains. For this example we are going to choose Ethereum
.
Raw Logs
.
Onchain Datasets
you can configure:
Start indexing at
: here you can define whether you want to do a full backfill (Beginning
) or read from edge (Current
)Filter by contract address
: optional contract address (in lowercase) to filter fromFilter by topics
: optional list of topics (in lowercase) to filter from, separated by commas.View Schema
: view the data schema to get a better idea of the shape of the data as well as see some sample records.+
button at the top right edge of the Data Source
card and you’ll have the option to add a Transform or a Sink.
Custom SQL
.
Query
field of the card to bring up the SQL editor.
SELECT id, block_number, transaction_hash, data FROM source_1
If you click on the Run
button on the top right corner you’ll see a preview of the final shape of the data. Once satisfied with the results in your Transforms, press Save
to add it to the pipeline.
+
button at the top right edge of the Transform Card
and select a Sink.
Preview Output
button to see the what the final shape of the data will be; this is a very convenient utility
in cases where you might have multiple sources and transforms in your pipeline and you want to iterate on its logic without having to redeploy the actual pipeline every time.
ethereum-raw-logs
Up to this point, your canvas should look similar to this:
Deploy
button on the top right corner and specify the resource size; for this example you can choose the default Small
.
Update Pipeline
button on the top right corner of the page and it will take you back onto the Flow canvas so you can do any updates on it.
YAML
on the top left corner. This is quite useful
in cases where you’d like to version control your pipeline logic and/or automate its deployment via CI/CD using the CLI (as explained in next section)source_1
; the transform (sql_1
) reads from source_1
in its SQL query; the sink (sink_1
)
reads the result from the transform (see its Input source
value) to finally emit the data into the destination. You can modify the reference names of every component of the pipeline on the canvas, just bear in mind
the connecting role these names play.Install Goldsky's CLI and log in
goldsky login
and paste your API key.
goldsky
to get started:
goldsky pipeline create <your-pipeline-name>
in your terminal and follow the prompts.
In short, the CLI guides you through the following process:
sources
and sinks
are required with a minimum of one entry each. transforms
is optional and an empty object ({}
) can be used if no transforms are needed.
Full configuration details for Pipelines is available in the reference page.
As an example, see below a pipeline configuration which uses the Ethereum Decoded Logs dataset as source, uses a transform to select specific data fields and sinks that data into a Postgres database whose connection details are stored within the A_POSTGRESQL_SECRET
secret:
Example pipeline configuration
goldsky pipeline apply <your-pipeline-config-file-path>
in your terminal to create a pipeline.
Once your pipeline is created, run goldsky pipeline start <your_pipeline_name>
to start your pipeline.
goldsky pipeline monitor <your-pipeline-name>
CLI command. It refreshes every ten seconds and gives you insights into how your pipeline performs.
Or you may monitor in the Pipeline Dashboard page at https://app.goldsky.com/dashboard/pipelines/stream/<pipeline_name>/<version>
where you can see the pipeline’s status
, logs
, metrics
.