> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Throttle

> Limit the throughput of a stream by batching and pacing records

## Overview

The throttle transform caps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Use it to:

* Stay under rate limits of downstream sinks or external APIs
* Smooth out bursty sources into a steady, predictable rate
* Test sink behavior at a controlled records-per-second rate
* Reduce pressure on small resource sizes during development

Throttle does not modify the data: every input record passes through unchanged. It only controls *when* records are emitted.

## Configuration

```yaml theme={null}
transforms:
  my_throttle:
    type: throttle
    from: <source-or-transform>
    max_batch_size: 100
    min_batch_interval: 10s
```

### Parameters

<ParamField path="type" type="string" required>
  Must be `throttle`
</ParamField>

<ParamField path="from" type="string" required>
  The source or transform to read data from
</ParamField>

<ParamField path="max_batch_size" type="integer" required>
  Maximum number of records to emit per batch. The throttle accumulates up to
  this many records before flushing.
</ParamField>

<ParamField path="min_batch_interval" type="duration" required>
  Minimum time to wait between batches (e.g., `10s`, `500ms`, `1m`). The next
  batch will not be emitted until this interval has elapsed since the previous
  batch.
</ParamField>

## How throttling works

Records are buffered as they arrive from the upstream source or transform. A batch is flushed downstream when **both** conditions are met:

* `max_batch_size` records have accumulated, **and**
* `min_batch_interval` has elapsed since the last batch was emitted

The effective maximum throughput is approximately:

```
max_batch_size / min_batch_interval = records per second
```

Examples:

* `max_batch_size: 100`, `min_batch_interval: 10s` → \~10 rps
* `max_batch_size: 1000`, `min_batch_interval: 5s` → \~200 rps

<Note>
  "Records per second" here means the **average throughput** the downstream
  system needs to handle, not literal requests or messages per second. The
  throttle emits one batch per interval: sinks consume that batch in whatever
  way is natural for them. For example, an S3 sink writes one file per
  interval at the configured batch size.
</Note>

<Note>
  Throttle limits the *maximum* rate, not the minimum. If the upstream is slow,
  batches will be smaller and arrive less frequently.
</Note>

## Example

Throttle a high-volume ERC-20 transfer stream down to \~10 rps before sending it to a sink:

```yaml theme={null}
name: throttle_example
resource_size: s
use_dedicated_ip: false
job: false

sources:
  erc20s:
    type: dataset
    dataset_name: matic.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  throttled_erc20s:
    type: throttle
    from: erc20s
    max_batch_size: 100 # ~10 rps with a 10s interval
    min_batch_interval: 10s

sinks:
  sink_1:
    type: blackhole
    from: throttled_erc20s
```

## When to use throttle

* **Rate-limited sinks**: Stay under per-second write quotas on downstream APIs or databases.
* **External handler protection**: Pace records into an [HTTP handler](/turbo-pipelines/transforms/http-handler) so the receiving service is not overwhelmed.
* **Cost control during development**: Slow down processing while iterating on a pipeline against a live source.
* **Testing**: Reproduce sink behavior under a known, fixed input rate.

## Best Practices

<Steps>
  <Step title="Place throttle close to the bottleneck">
    Throttle the stream just before the rate-limited sink or handler so
    upstream transforms still process at full speed.
  </Step>

  <Step title="Tune batch size to your sink">
    Larger `max_batch_size` reduces per-batch overhead but increases latency
    per record. Pick a size that matches your sink's preferred batch size.
  </Step>

  <Step title="Remove throttle in production where possible">
    Throttle caps throughput by design. Once rate-limit concerns are addressed,
    remove the transform to let the pipeline run at full speed.
  </Step>
</Steps>