Sync dataset to Postgres

This pipeline is named raw-logs-pipeline. It pulls data from a Goldsky dataset, selects a handful of columns without any further transformation, and stores the result in a PostgreSQL sink, in a table called eth_logs in the goldsky schema.

raw-logs-pipeline.yaml

name: raw-logs-pipeline
resource_size: s

sources:
  my_ethereum_raw_logs:
    type: dataset
    dataset_name: ethereum.raw_logs
    version: 1.0.0
    start_at: latest

transforms:
  # Project the log columns we want to persist
  logs:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        address,
        topics,
        data,
        block_number,
        block_hash,
        transaction_hash
      FROM my_ethereum_raw_logs

sinks:
  logs_sink:
    type: postgres
    from: logs
    schema: goldsky
    table: eth_logs
    secret_name: API_POSTGRES_CREDENTIALS
    primary_key: id

The secret_name references a secret containing your Postgres connection string (postgres://username:password@host:port/database). Create one with goldsky secret create API_POSTGRES_CREDENTIALS; see managing secrets. You can start the above pipeline by running:

goldsky turbo apply raw-logs-pipeline.yaml

Once it’s running, watch the records flow through it with live inspect:

goldsky turbo inspect raw-logs-pipeline

Turbo

Guides

Reference

Mirror (legacy)

Sync dataset to Postgres