Data warehouses - Goldsky Docs

Overview

You can get Goldsky data into any data warehouse that can ingest from Amazon S3. Turbo pipelines write to S3 with the S3 sink, and your warehouse loads those files with its built-in S3 ingestion. You don’t need a third-party connector such as Fivetran. The S3 sink writes standard Apache Parquet files with optional Hive-style partitioning. Every major warehouse reads Parquet from S3 natively, so this pattern works with Snowflake, Databricks, BigQuery, Redshift, and others.

Goldsky doesn’t have a native sink for these warehouses. S3 acts as the intermediary. Batch loading from S3 is often cheaper than streaming into a warehouse, because streaming inserts typically require compute that is always on.

How it works

A Turbo pipeline writes your data to an S3 bucket you own as Parquet files, using the S3 sink.
Your warehouse reads those files with its own S3 ingestion feature, either continuously or on a schedule.
You query the data in your warehouse alongside the rest of your analytics.

Supported data warehouses

Any warehouse or query engine that reads Parquet from S3 fits this pattern. Popular options include:

Data warehouse	How it ingests from S3
Snowflake	Snowpipe for continuous loading, `COPY INTO` from an external stage, or external tables
Databricks	Auto Loader for incremental ingestion, `COPY INTO`, or external tables in Unity Catalog
Google BigQuery	BigQuery Data Transfer Service for Amazon S3, or BigQuery Omni
Amazon Redshift	`COPY` command, auto-copy from S3, or Redshift Spectrum external tables
Amazon Athena	External tables queried in place over S3
Microsoft Fabric	OneLake shortcuts to Amazon S3
Trino and Starburst	Hive or Iceberg connectors over S3
Dremio	S3 data source
DuckDB and MotherDuck	`read_parquet` over `s3://` paths

Refer to your warehouse’s documentation for the exact setup, including how to grant it read access to your bucket.

Set up the pipeline

Configure an S3 sink that writes Parquet to a bucket your warehouse can read. Partitioning by date (dt) lets your warehouse load new data incrementally and prune scans.

name: erc20-to-warehouse
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  enriched:
    type: sql
    primary_key: id
    sql: |
      SELECT
        *,
        CAST(block_timestamp AS DATE) AS dt
      FROM transfers

sinks:
  warehouse_staging:
    type: s3_sink
    from: enriched
    bucket: my-bucket
    prefix: eth/erc20_transfers
    partition_columns: dt
    compression: zstd
    secret_name: MY_S3_SECRET

See the S3 sink page for all parameters, the secret format, the file layout, and required IAM permissions.

Load into your warehouse

The examples below show the general shape of a load. Adapt names, credentials, and access configuration to your environment.

Snowflake
Databricks

Create an external stage that points at the sink’s prefix, then copy the Parquet files into a table. Run the COPY INTO on a schedule, or wrap it in a Snowpipe for continuous loading.

CREATE STAGE goldsky_erc20_stage
  URL = 's3://my-bucket/eth/erc20_transfers/'
  STORAGE_INTEGRATION = my_s3_integration
  FILE_FORMAT = (TYPE = PARQUET);

COPY INTO erc20_transfers
  FROM @goldsky_erc20_stage
  MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

For a worked example of scheduling S3 loads into Snowflake with Python, see the Snowflake guide.

Use COPY INTO to load new Parquet files idempotently. Databricks tracks which files it has already loaded, so you can rerun the command on a schedule. Auto Loader is an alternative for streaming ingestion.

CREATE TABLE IF NOT EXISTS main.goldsky.erc20_transfers;

COPY INTO main.goldsky.erc20_transfers
  FROM 's3://my-bucket/eth/erc20_transfers/'
  FILEFORMAT = PARQUET
  COPY_OPTIONS ('mergeSchema' = 'true');

Best practices

Deduplicate on the primary key

Turbo sinks deliver data at least once, so a restart can write the same rows to S3 again. Deduplicate on the transform’s primary_key (for example id) in your warehouse, either with a MERGE during loading or in a downstream model.

Partition by date

A dt partition column lets your warehouse load only new partitions and skip old files. Add more partition columns only when your queries consistently filter on them, because too many partitions produce many small files.

Use zstd compression

Set compression: zstd on the S3 sink to reduce storage and transfer costs. Check that your warehouse supports zstd-compressed Parquet; Snowflake and Databricks both do.

Turbo

Guides

Reference

Mirror (legacy)

Data warehouses (Snowflake, Databricks, BigQuery, and more)

Overview

How it works

Supported data warehouses

Set up the pipeline

Load into your warehouse

Best practices

​Overview

​How it works

​Supported data warehouses

​Set up the pipeline

​Load into your warehouse

​Best practices

Overview

How it works

Supported data warehouses

Set up the pipeline

Load into your warehouse

Best practices