Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt

Use this file to discover all available pages before exploring further.

A Webhook sink allows you to send data to an external service via HTTP. This provides considerable flexibility for forwarding pipeline results to your application server, a third-party API, or a bot. Webhook sinks ensure at least once delivery and manage back-pressure, meaning data delivery adapts based on the responsiveness of your endpoints. The pipeline sends a POST request with a JSON payload to a specified URL, and the receiver only needs to return a 200 status code to confirm successful delivery. Here is a snippet of YAML that specifies a Webhook sink:

​
Pipeline configuration

sinks:
  my_webhook_sink:
    type: webhook
    
    # The webhook url
    url: Type.String()

    # The object key coming from either a source or transform. 
    # Example: ethereum.raw_blocks.
    from: Type.String()

    # The name of a goldsky httpauth secret you created which contains a header that can be used for authentication. More on how to create these in the section below.
    secret_name: Type.Optional(Type.String())

    # Optional metadata that you want to send on every request.
    headers:
      SOME-HEADER-KEY: Type.Optional(Type.String())
    
    # Whether to send only one row per HTTP request (better for compatibility with third-party integrations - e.g bots) or to mini-batch it (better for throughput).  
    one_row_per_request: Type.Optional(Type.Boolean())

    # The number of records the sink will send together in a batch. Default `100`
    batch_size: Type.Optional(Type.Integer())

    # The maximum time the sink will batch records before flushing. Examples: 60s, 1m, 1h. Default: '1s'
    batch_flush_interval: Type.Optional(Type.String())

​
Key considerations

  • Failure Handling: In case of failures, the pipeline retries requests indefinitely with exponential backoff.
  • Networking & Performance: For optimal performance, deploy your webhook server in a region close to where the pipelines are deployed (we use aws us-west-2). Aim to keep p95 latency under 100 milliseconds for best results.
  • Latency vs Throughput: Use lower batch_size/batch_flush_interval to achive low latency and higher values to achieve high throughput (useful when backfilling/bootstraping).
  • Connection & Response times: The maximum allowed response time is 5 minutes and the maximum allowed time to establish a connection is 1 minute.

​
Secret creation

Create a httpauth secret with the following CLI command: 
goldsky secret create
Select httpauth as the secret type and then follow the prompts to finish creating your httpauth secret.

​
Example Webhook sink configuration

sinks:
  my_webhook_sink:
    type: webhook
    url: https://my-webhook-service.com/webhook-1
    from: ethereum.raw_blocks
    secret_name: ETH_BLOCKS_SECRET