Guides
Sync dataset to Postgres
Mirror
- Introduction
- Getting started
- About Mirror Pipelines
- Data Sources
- Transforming Data
- Data Sinks
Extensions
- Overview
- Channels
More Info
Guides
Sync dataset to Postgres
This pipeline is named decoded-logs-pipeline
. It pulls data from a curated goldsky dataset, without performing any transformations, and stores the result into a PostgreSQL sink, in a table called eth_logs
in the goldsky
schema.
decoded-logs-pipeline.yaml
name: decoded-logs-pipeline
apiVersion: 3
sources:
my_ethereum_decoded_logs:
dataset_name: ethereum.decoded_logs
version: 1.0.0
type: dataset
start_at: latest
transforms:
logs:
sql: |
SELECT
id,
address,
event_signature,
event_params,
raw_log.block_number as block_number,
raw_log.block_hash as block_hash,
raw_log.transaction_hash as transaction_hash
FROM
my_ethereum_decoded_logs
primary_key: id
sinks:
logs_sink:
type: postgres
table: eth_logs
schema: goldsky
secret_name: API_POSTGRES_CREDENTIALS
from: logs
decoded-logs-pipeline.yaml
name: decoded-logs-pipeline
apiVersion: 3
sources:
my_ethereum_decoded_logs:
dataset_name: ethereum.decoded_logs
version: 1.0.0
type: dataset
start_at: latest
transforms:
logs:
sql: |
SELECT
id,
address,
event_signature,
event_params,
raw_log.block_number as block_number,
raw_log.block_hash as block_hash,
raw_log.transaction_hash as transaction_hash
FROM
my_ethereum_decoded_logs
primary_key: id
sinks:
logs_sink:
type: postgres
table: eth_logs
schema: goldsky
secret_name: API_POSTGRES_CREDENTIALS
from: logs
sources:
- referenceName: ethereum.decoded_logs
version: 1.0.0
type: dataset
startAt: latest
transforms:
- sql: |
SELECT
id,
address,
event_signature,
event_params,
raw_log.block_number as block_number,
raw_log.block_hash as block_hash,
raw_log.transaction_hash as transaction_hash
FROM
ethereum.decoded_logs
referenceName: logs
type: sql
primaryKey: id
sinks:
- type: postgres
table: eth_logs
schema: goldsky
secretName: API_POSTGRES_CREDENTIALS
sourceStreamName: logs
referenceName: logs_sink
You can start above pipeline by running:
goldsky pipeline start pipeline.yaml
Or
goldsky pipeline apply pipeline.yaml --status ACTIVE
Was this page helpful?