Sync dataset to Postgres

This pipeline is named raw-logs-pipeline. It pulls data from a Goldsky dataset, without performing any transformations, and stores the result into a PostgreSQL sink, in a table called eth_logs in the goldsky schema.

raw-logs-pipeline.yaml

name: raw-logs-pipeline
apiVersion: 3
sources:
  my_ethereum_raw_logs:
    dataset_name: ethereum.raw_logs
    version: 1.0.0
    type: dataset
    start_at: latest

transforms:
  logs:
    sql: |
      SELECT
          id,
          address,
          topics,
          data,
          block_number,
          block_hash,
          transaction_hash
      FROM
          my_ethereum_raw_logs
    primary_key: id

sinks:
  logs_sink:
    type: postgres
    table: eth_logs
    schema: goldsky
    secret_name: API_POSTGRES_CREDENTIALS
    from: logs

You can start above pipeline by running:

goldsky pipeline start pipeline.yaml

goldsky pipeline apply pipeline.yaml --status ACTIVE