A Webhook sink allows you to send data to an external service via HTTP. This provides considerable flexibility for forwarding pipeline results to your application server, a third-party API, or a bot.Webhook sinks ensure at least once delivery and manage back-pressure, meaning data delivery adapts based on the responsiveness of your endpoints. The pipeline sends a POST request with a JSON payload to a specified URL, and the receiver only needs to return a 200 status code to confirm successful delivery.Here is a snippet of YAML that specifies a Webhook sink:
sinks: my_webhook_sink: type: webhook # The webhook url url: Type.String() # The object key coming from either a source or transform. # Example: ethereum.raw_blocks. from: Type.String() # The name of a goldsky httpauth secret you created which contains a header that can be used for authentication. More on how to create these in the section below. secret_name: Type.Optional(Type.String()) # Optional metadata that you want to send on every request. headers: SOME-HEADER-KEY: Type.Optional(Type.String()) # Whether to send only one row per http request (better for compatibility with third-party integrations - e.g bots) or to mini-batch it (better for throughput). one_row_per_request: Type.Optional(Type.Boolean()) # The number of records the sink will send together in a batch. Default `100` batch_size: Type.Optional(Type.Integer()) # The maximum time the sink will batch records before flushing. Examples: 60s, 1m, 1h. Default: '1s' batch_flush_interval: Type.Optional(Type.String())
Failure Handling: In case of failures, the pipeline retries requests indefinitely with exponential backoff.
Networking & Performance: For optimal performance, deploy your webhook server in a region close to where the pipelines are deployed (we use aws us-west-2). Aim to keep p95 latency under 100 milliseconds for best results.
Latency vs Throughput: Use lower batch_size/batch_flush_interval to achive low latency and higher values to achieve high throughput (useful when backfilling/bootstraping).
Connection & Response times: The maximum allowed response time is 5 minutes and the maximum allowed time to establish a connection is 1 minute.