Webhook

A Webhook sink allows you to send data to an external service via HTTP. This provides considerable flexibility for forwarding pipeline results to your application server, a third-party API, or a bot. Webhook sinks ensure at least once delivery and manage back-pressure, meaning data delivery adapts based on the responsiveness of your endpoints. The pipeline sends a POST request with a JSON payload to a specified URL, and the receiver only needs to return a 200 status code to confirm successful delivery.

Turbo’s webhook sink supports the same delivery semantics and adds a primary_key field for sink-side checkpointing, one_row_per_request, and payload_version envelope formats.

Here is a snippet of YAML that specifies a Webhook sink:

Pipeline configuration

sinks:
  my_webhook_sink:
    type: webhook
    
    # The webhook url
    url: Type.String()

    # The object key coming from either a source or transform. 
    # Example: ethereum.raw_blocks.
    from: Type.String()

    # The name of a goldsky httpauth secret you created which contains a header that can be used for authentication. More on how to create these in the section below.
    secret_name: Type.Optional(Type.String())

    # Optional metadata that you want to send on every request.
    headers:
      SOME-HEADER-KEY: Type.Optional(Type.String())
    
    # Whether to send only one row per HTTP request (better for compatibility with third-party integrations - e.g bots) or to mini-batch it (better for throughput).  
    one_row_per_request: Type.Optional(Type.Boolean())

    # The number of records the sink will send together in a batch. Default `100`
    batch_size: Type.Optional(Type.Integer())

    # The maximum time the sink will batch records before flushing. Examples: 60s, 1m, 1h. Default: '1s'
    batch_flush_interval: Type.Optional(Type.String())

Key considerations

Failure Handling: In case of failures, the pipeline retries requests indefinitely with exponential backoff.
Networking & Performance: For optimal performance, deploy your webhook server in a region close to where the pipelines are deployed (we use aws us-west-2). Aim to keep p95 latency under 100 milliseconds for best results.
Latency vs Throughput: Use lower batch_size/batch_flush_interval to achive low latency and higher values to achieve high throughput (useful when backfilling/bootstraping).
Connection & Response times: The maximum allowed response time is 5 minutes and the maximum allowed time to establish a connection is 1 minute.

Secret creation

Create a httpauth secret with the following CLI command:

goldsky secret create

Select httpauth as the secret type and then follow the prompts to finish creating your httpauth secret.

Example Webhook sink configuration

sinks:
  my_webhook_sink:
    type: webhook
    url: https://my-webhook-service.com/webhook-1
    from: ethereum.raw_blocks
    secret_name: ETH_BLOCKS_SECRET

Mirror

Guides

Reference

Pipeline configuration

Key considerations

Secret creation

Example Webhook sink configuration

​Pipeline configuration

​Key considerations

​Secret creation

​Example Webhook sink configuration

Pipeline configuration

Key considerations

Secret creation

Example Webhook sink configuration