How to throttle streaming data throughput in Turbo pipelines

Overview

The throttle transform caps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Use it to:

Stay under rate limits of downstream sinks or external APIs
Smooth out bursty sources into a steady, predictable rate
Test sink behavior at a controlled records-per-second rate
Reduce pressure on small resource sizes during development

Throttle does not modify the data: every input record passes through unchanged. It only controls when records are emitted.

Configuration

transforms:
  my_throttle:
    type: throttle
    from: <source-or-transform>
    max_batch_size: 100
    min_batch_interval: 10s

Parameters

string

required

Must be throttle

string

required

The source or transform to read data from

integer

default:"100"

Row budget per min_batch_interval. Must be greater than 0. Defaults to 100.

duration

default:"1s"

Minimum time between emissions, in humantime format (e.g., 500ms, 1s, 2m). Must be greater than 0. Defaults to 1s.

Invalid values (max_batch_size: 0, an unparseable duration, etc.) fail at pipeline start with a configuration error.

How throttling works

Batches arrive from the upstream source or transform and pass through unchanged. The throttle paces when each batch is released so that the long-run row rate does not exceed max_batch_size rows per min_batch_interval. For each incoming batch of rows rows:

If the schedule says the throttle is not yet eligible to emit, it sleeps until it is.
The batch is emitted downstream, unchanged.

The next-eligible time advances by:

cost = max(min_batch_interval, ceil(rows / max_batch_size) * min_batch_interval)

Batches are never split and never dropped. An oversized batch passes through whole and then “pays” for itself on the schedule, so the long-run rate is preserved. The effective maximum throughput is approximately:

max_batch_size / min_batch_interval = rows per second

Examples with max_batch_size: 100, min_batch_interval: 1s:

Incoming batch sizes	Behavior
`50, 50, 50, …`	One batch per second. Average rate: 50 rows/s.
`100, 100, 100, …`	One batch per second. Average rate: 100 rows/s.
`1000, 1, 1, …`	First emits immediately; the next is delayed 10s.
`200, 1, 1, …`	First emits immediately; the next is delayed 2s.
Idle for 1 minute, then `50`	Emits immediately (a long idle resets the schedule).

“Rows per second” here is the average throughput the downstream system needs to handle, not literal requests or messages per second. The throttle emits one batch per interval; sinks consume that batch in whatever way is natural for them. For example, an S3 sink writes one file per interval at the configured batch size.

Throttle limits the maximum rate, not the minimum. If the upstream is slow, batches will be smaller and arrive less frequently. Empty batches pass through immediately and do not advance the schedule.

Backpressure

When the throttle is sleeping, its input slot fills up and the upstream stage’s send blocks. That stall walks all the way back to the source, so no batches are dropped and in-flight memory stays bounded. Shutdown is not delayed by a long min_batch_interval — an in-flight sleep is cancelled when the pipeline terminates.

Example

Throttle a high-volume ERC-20 transfer stream down to ~10 rps before sending it to a sink:

name: throttle_example
resource_size: s
use_dedicated_ip: false
job: false

sources:
  erc20s:
    type: dataset
    dataset_name: matic.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  throttled_erc20s:
    type: throttle
    from: erc20s
    max_batch_size: 100 # ~10 rps with a 10s interval
    min_batch_interval: 10s

sinks:
  sink_1:
    type: blackhole
    from: throttled_erc20s

When to use throttle

Rate-limited sinks: Stay under per-second write quotas on downstream APIs or databases.
External handler protection: Pace records into an HTTP handler so the receiving service is not overwhelmed.
Cost control during development: Slow down processing while iterating on a pipeline against a live source.
Testing: Reproduce sink behavior under a known, fixed input rate.

Best Practices

Place throttle close to the bottleneck

Throttle the stream just before the rate-limited sink or handler so upstream transforms still process at full speed.

Tune batch size to your sink

Larger max_batch_size reduces per-batch overhead but increases latency per record. Pick a size that matches your sink’s preferred batch size.

Remove throttle in production where possible

Throttle caps throughput by design. Once rate-limit concerns are addressed, remove the transform to let the pipeline run at full speed.

​Overview

​Configuration

​Parameters

​How throttling works

​Backpressure

​Example

​When to use throttle

​Best Practices

Overview

Configuration

Parameters

How throttling works

Backpressure

Example

When to use throttle

Best Practices