> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data warehouses (Snowflake, Databricks, BigQuery, and more)

> Load Turbo pipeline data into Snowflake, Databricks, BigQuery, Redshift, or any data warehouse that can ingest Parquet files from Amazon S3.

## Overview

You can get Goldsky data into any data warehouse that can ingest from Amazon S3. Turbo pipelines write to S3 with the [S3 sink](https://docs.goldsky.com/turbo-pipelines/sinks/s3.md), and your warehouse loads those files with its built-in S3 ingestion. You don't need a third-party connector such as Fivetran.

The S3 sink writes standard Apache Parquet files with optional Hive-style partitioning. Every major warehouse reads Parquet from S3 natively, so this pattern works with Snowflake, Databricks, BigQuery, Redshift, and others.

> [!NOTE]
>
> Goldsky doesn't have a native sink for these warehouses. S3 acts as the
> intermediary. Batch loading from S3 is often cheaper than streaming into a
> warehouse, because streaming inserts typically require compute that is
> always on.

## How it works

1. A Turbo pipeline writes your data to an S3 bucket you own as Parquet files, using the [S3 sink](https://docs.goldsky.com/turbo-pipelines/sinks/s3.md).
2. Your warehouse reads those files with its own S3 ingestion feature, either continuously or on a schedule.
3. You query the data in your warehouse alongside the rest of your analytics.

## Supported data warehouses

Any warehouse or query engine that reads Parquet from S3 fits this pattern. Popular options include:

| Data warehouse | How it ingests from S3 |
| - | - |
| **Snowflake** | Snowpipe for continuous loading, `COPY INTO` from an external stage, or external tables |
| **Databricks** | Auto Loader for incremental ingestion, `COPY INTO`, or external tables in Unity Catalog |
| **Google BigQuery** | BigQuery Data Transfer Service for Amazon S3, or BigQuery Omni |
| **Amazon Redshift** | `COPY` command, auto-copy from S3, or Redshift Spectrum external tables |
| **Amazon Athena** | External tables queried in place over S3 |
| **Microsoft Fabric** | OneLake shortcuts to Amazon S3 |
| **Trino and Starburst** | Hive or Iceberg connectors over S3 |
| **Dremio** | S3 data source |
| **DuckDB and MotherDuck** | `read_parquet` over `s3://` paths |

Refer to your warehouse's documentation for the exact setup, including how to grant it read access to your bucket.

## Set up the pipeline

Configure an S3 sink that writes Parquet to a bucket your warehouse can read. Partitioning by date (`dt`) lets your warehouse load new data incrementally and prune scans.

```yaml theme={"dark"}
name: erc20-to-warehouse
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  enriched:
    type: sql
    primary_key: id
    sql: |
      SELECT
        *,
        CAST(block_timestamp AS DATE) AS dt
      FROM transfers

sinks:
  warehouse_staging:
    type: s3_sink
    from: enriched
    bucket: my-bucket
    prefix: eth/erc20_transfers
    partition_columns: dt
    compression: zstd
    secret_name: MY_S3_SECRET
```

See the [S3 sink](https://docs.goldsky.com/turbo-pipelines/sinks/s3.md) page for all parameters, the secret format, the file layout, and required IAM permissions.

## Load into your warehouse

The examples below show the general shape of a load. Adapt names, credentials, and access configuration to your environment.

> [tab] Snowflake
>
> Create an external stage that points at the sink's prefix, then copy the
> Parquet files into a table. Run the `COPY INTO` on a schedule, or wrap it
> in a Snowpipe for continuous loading.
>
> ```sql theme={"dark"}
> CREATE STAGE goldsky_erc20_stage
>   URL = 's3://my-bucket/eth/erc20_transfers/'
>   STORAGE_INTEGRATION = my_s3_integration
>   FILE_FORMAT = (TYPE = PARQUET);
>
> COPY INTO erc20_transfers
>   FROM @goldsky_erc20_stage
>   MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
> ```
>
> For a worked example of scheduling S3 loads into Snowflake with Python, see the [Snowflake guide](https://docs.goldsky.com/mirror/sinks/snowflake.md).

> [tab] Databricks
>
> Use `COPY INTO` to load new Parquet files idempotently. Databricks tracks
> which files it has already loaded, so you can rerun the command on a
> schedule. Auto Loader is an alternative for streaming ingestion.
>
> ```sql theme={"dark"}
> CREATE TABLE IF NOT EXISTS main.goldsky.erc20_transfers;
>
> COPY INTO main.goldsky.erc20_transfers
>   FROM 's3://my-bucket/eth/erc20_transfers/'
>   FILEFORMAT = PARQUET
>   COPY_OPTIONS ('mergeSchema' = 'true');
> ```

## Best practices

> [accordion] Deduplicate on the primary key
>
> Turbo sinks deliver data at least once, so a restart can write the same
> rows to S3 again. Deduplicate on the transform's `primary_key` (for
> example `id`) in your warehouse, either with a `MERGE` during loading or
> in a downstream model.

> [accordion] Partition by date
>
> A `dt` partition column lets your warehouse load only new partitions and
> skip old files. Add more partition columns only when your queries
> consistently filter on them, because too many partitions produce many
> small files.

> [accordion] Use zstd compression
>
> Set `compression: zstd` on the S3 sink to reduce storage and transfer
> costs. Check that your warehouse supports zstd-compressed Parquet; Snowflake
> and Databricks both do.


## Related topics

- [Turbo - Supported sinks](https://docs.goldsky.com/turbo-pipelines/sinks/overview.md)
- [Which product do I need?](https://docs.goldsky.com/which-product.md)
- [PostgreSQL](https://docs.goldsky.com/turbo-pipelines/sinks/postgres.md)
- [Stream onchain data with Turbo](https://docs.goldsky.com/turbo-pipelines/introduction.md)
- [TypeScript transforms](https://docs.goldsky.com/turbo-pipelines/transforms/typescript.md)