Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Write data to PostgreSQL databases with automatic table creation and upsert support. Compatible with managed Postgres providers like Neon, Supabase, and AWS RDS.

​
Configuration

sinks:
  my_postgres_sink:
    type: postgres
    from: my_transform
    schema: public
    table: my_table
    secret_name: MY_POSTGRES_SECRET
    primary_key: id # Optional - enables upsert behavior

​
Parameters

​
type
string
required
Must be postgres.
​
from
string
required
Name of the transform or source to read data from.
​
schema
string
required
PostgreSQL schema name (for example, public, analytics). Created automatically if it doesn’t exist.
​
table
string
required
Table name to write to. Created automatically if it doesn’t exist, with columns inferred from the upstream Arrow schema.
​
secret_name
string
required
Name of a Goldsky secret holding the Postgres connection details (host, port, user, password, databaseName). See Secret format.
​
primary_key
string
Comma-separated list of columns to use as the table’s primary key. When set, writes become upserts (INSERT ... ON CONFLICT DO UPDATE). When omitted, writes are plain inserts and no primary key is added to the auto-created table.
​
on_conflict
string
default:"update"
Behavior when a row with the same primary_key already exists. update runs ON CONFLICT ... DO UPDATE SET, nothing runs ON CONFLICT ... DO NOTHING. Only applies when primary_key is set.
​
batch_size
integer
Maximum number of rows to accumulate before flushing to Postgres. Falls back to the engine default when unset.
​
batch_flush_interval
string
Maximum time to wait before flushing a partial batch, parsed as a humantime duration (for example, "500ms", "1s", "2s"). Falls back to the engine default when unset.
​
parallelism
integer
default:"1"
Number of parallel writer tasks. Each task processes a slice of the accumulated batch concurrently. Increase for sink-bound pipelines.

​
Secret format

Create the Postgres secret with the Goldsky CLI. The CLI accepts a standard Postgres connection string and stores it as a JDBC-style secret (host, port, user, password, databaseName): 
goldsky secret create MY_POSTGRES_SECRET
When prompted, paste a connection string such as: 
postgres://username:password@host:port/database
For provider-specific guidance (Neon, Supabase, RDS), see the secrets skill.

​
Behavior

  • Auto schema and table creation: Both the schema and the table are created with CREATE ... IF NOT EXISTS on first write. Columns are derived from the upstream Arrow schema.
  • No automatic schema migrations: If the table already exists, the engine does not ALTER TABLE to add or change columns. Upstream schema changes that add new columns will fail at insert time — drop or manually migrate the table first.
  • Upsert vs. insert: Setting primary_key produces INSERT ... ON CONFLICT (<pk>) DO UPDATE SET .... Without primary_key, rows are plain INSERTs and no primary key constraint is created.
  • Type mapping: Arrow types are mapped automatically — Int32 → INTEGER, Int64 → BIGINT, Utf8 → TEXT, Boolean → BOOLEAN, Decimal(p, s) → NUMERIC(p, s), structs/lists/maps → JSONB.
  • Large numbers: Blockchain U256/I256 values and UInt64 are stored as NUMERIC.

​
Example

sinks:
  postgres_transfers:
    type: postgres
    from: filtered_transfers
    schema: public
    table: erc20_transfers
    secret_name: MY_POSTGRES
    primary_key: id