Turbo - Supported sinks

Overview

Sinks are the final destination for data in your Turbo pipelines. They write processed data to external systems like databases, data warehouses, or HTTP endpoints.

Available Sinks

PostgreSQL

Write to PostgreSQL databases

PostgreSQL aggregation

Real-time aggregations in PostgreSQL

ClickHouse

Write to ClickHouse for analytics

MySQL

Write to MySQL databases

Webhook

Send data to HTTP endpoints

Kafka

Publish to Kafka topics

S3

Write to S3-compatible storage

SQS

Send to Amazon SQS queues

Google Cloud Pub/Sub

Publish to Google Cloud Pub/Sub topics

S2

Publish to S2.dev streams

Blackhole

Discard data for testing purposes

Common Parameters

Every sink accepts at least these parameters. See each sink’s page for type-specific fields.

string

required

The sink type — for example postgres, clickhouse, kafka, webhook, mysql_sink, s3_sink, sqs_sink, pubsub, s2_sink, blackhole.

string

required

Name of the transform or source to read data from.

string

Name of the Goldsky secret containing connection credentials. Recommended for any sink that connects to an external system (databases, Kafka, S3, SQS, webhooks, etc.). Some sinks also accept inline credentials — see each sink’s page for details.

string

Column (or comma-separated list of columns) used to identify unique rows. Required for upserts in database sinks and required (not optional) on the ClickHouse sink.

integer

Maximum number of rows the Turbo runtime buffers before flushing a batch to the sink. Applies to every sink — batching is a property of the runtime, not of the individual sink. Individual sink pages may document defaults or upper limits that make sense for that destination (for example the Kafka sink also exposes librdkafka-level batch_size).

string

Maximum time the Turbo runtime waits before flushing a batch, even if batch_size has not been reached. Applies to every sink. Accepts humantime durations such as 500ms, 1s, or 30s. A bare number is interpreted as milliseconds (for example "1000" is equivalent to 1s).

Batching

Every Turbo sink accepts batch_size and batch_flush_interval — they are handled by the Turbo runtime, so they work on every sink even when a specific sink’s page does not list them explicitly. Rows accumulate in an in-memory buffer and flush to the sink whenever either threshold is reached, whichever comes first.

Larger batches — increase throughput and reduce write amplification, at the cost of higher end-to-end latency.
Smaller batches / shorter intervals — reduce latency, useful when a downstream system needs to react to rows quickly.
Defaults — batch_size defaults to 1000 rows and batch_flush_interval defaults to 1s. Individual sinks may override these; see each sink’s page for destination-specific guidance.

Multiple Sinks

You can write the same data to multiple destinations:

transforms:
  processed_data:
    type: sql
    primary_key: id
    sql: SELECT * FROM source

sinks:
  # Write to PostgreSQL
  postgres_archive:
    type: postgres
    from: processed_data
    schema: public
    table: archive
    secret_name: MY_POSTGRES

  # Send to webhook
  webhook_notification:
    type: webhook
    from: processed_data
    url: https://api.example.com/notify
    secret_name: NOTIFY_API_SECRET

  # Publish to Kafka
  kafka_downstream:
    type: kafka
    from: processed_data
    topic: processed.events

Each sink writes its own copy of the data. Because all sinks share the same checkpoint barrier, a sink that is slow or stalled will apply backpressure to the whole pipeline (see below).

Sink Behavior

Checkpointing

All sinks participate in Turbo’s checkpointing system:

Each sink buffers writes and acknowledges only after it has flushed them
Sources only commit their position after every sink has acknowledged
This gives at-least-once delivery — duplicates are possible after a restart, but no committed data is lost

Backpressure

Sinks apply backpressure to the pipeline:

If any sink can’t keep up, the entire pipeline slows down
Prevents data loss and memory overflow
Monitor sink performance to identify bottlenecks

Error Handling

Sink errors are retried with exponential backoff. Transient errors (network blips, connection resets) recover automatically. Persistent errors (bad credentials, schema mismatches, unreachable destination) will keep retrying and show up as stalled checkpoints — check the pipeline logs for the underlying error message.

Best Practices

Choose the right sink for your use case

PostgreSQL: Transactional data, updates, relational queries
PostgreSQL aggregation: Real-time aggregations like balances, totals, counts
ClickHouse: High-volume analytics, aggregations, time-series
MySQL: Transactional data, updates, relational queries
Webhook: Real-time notifications, integrations with external systems
Kafka: Downstream processing, event sourcing, decoupling systems
SQS: Event-driven architectures, decoupled integrations, message queuing
Pub/Sub: Fan-out to many GCP consumers, event-driven workloads on Google Cloud
S2: Decoupled processing, large number of readers, serverless architectures

Use appropriate primary keys

For upsert behavior in databases, choose a stable primary key:

sinks:
  postgres_sink:
    primary_key: id  # Use a unique, stable identifier

Monitor sink performance

Use logs and metrics to track:

Write throughput
Error rates
Latency

goldsky turbo logs my-pipeline

Secure credentials

Always use secrets for database credentials:

# Create secret
goldsky secret create MY_DB_SECRET

# Reference in pipeline
sinks:
  my_sink:
    secret_name: MY_DB_SECRET

​Overview

​Available Sinks

PostgreSQL

PostgreSQL aggregation

ClickHouse

MySQL

Webhook

Kafka

S3

SQS

Google Cloud Pub/Sub

S2

Blackhole

​Common Parameters

​Batching

​Multiple Sinks

​Sink Behavior

​Checkpointing

​Backpressure

​Error Handling

​Best Practices

Overview

Available Sinks

Common Parameters

Batching

Multiple Sinks

Sink Behavior

Checkpointing

Backpressure

Error Handling

Best Practices