External Handler Transforms
Transforming data with an external http service.
With external handler transforms, you can send data from your Mirror pipeline to an external service via HTTP and return the processed results back into the pipeline. This opens up a world of possibilities by allowing you to bring your own custom logic, programming languages, and external services into the transformation process.
In this repo you can see an example implementation of enriching ERC-20 Transfer Events with an HTTP service.
Key Features of External Handler Transforms:
- Send data to external services via HTTP.
- Supports a wide variety of programming languages and external libraries.
- Handle complex processing outside the pipeline and return results in real time.
- Guaranteed at least once delivery and back-pressure control to ensure data integrity.
How External Handlers work
- The pipeline sends a POST request to the external handler with a mini-batch of JSON rows.
- The external handler processes the data and returns the transformed rows in the same format and order as received.
Example workflow
- The pipeline sends data to an external service (e.g. a custom API).
- The service processes the data and returns the results to the pipeline.
- The pipeline continues processing the enriched data downstream.
Example HTTP Request
Example HTTP Response
YAML config with an external transform
Schema override datatypes
When overriding the schema of the data returned by the handler it’s important to get the datatypes for each column right. The schema_override property is a map of column names to Flink SQL datatypes.
Key considerations
- Schema Changes: If the external handler’s output schema changes, you will need to redeploy the pipeline with the relevant schema_override.
- Failure Handling: In case of failures, the pipeline retries requests indefinitely with exponential backoff.
- Networking & Performance: For optimal performance, deploy your handler in a region close to where the pipelines are deployed (we use aws
us-west-2
). Aim to keep p95 latency under 100 milliseconds for best results. - Connection & Response times: The maximum allowed response time is 5 minutes and the maximum allowed time to establish a connection is 1 minute.
In-order mode for external handlers
In-Order mode allows for subgraph-style processing inside mirror. Records are emitted to the handler in the order that they appear on-chain.
How to get started
- Make sure that the sources that you want to use currently support Fast Scan. If they don’t, submit a request to support.
- In your pipeline definition specify the
filter
andin_order
attributes for your source. - Declare a transform of type handler or a sink of type webhook.
Simple transforms (e.g filtering) in between the source and the handler/webhook are allowed, but other complex transforms (e.g. aggregations, joins) can cause loss of ordering.
Example YAML config, with in-order mode
Example in-order webhook sink
In-order mode tips
- To observe records in order, either have a single instance of your handler responding to requests OR introduce some coordination mechanism to make sure that only one replica of the service can answer at a time.
- When deploying your service, avoid having old and new instances running at the same time. Instead, discard the current instance and incur a little downtime to preserve ordering.
- When receiving messages that have already been processed in the handler (pre-existing idempotency key or previous index (e.g already seen block number)) don’t introduce any side effects on your side, but do respond to the message as usual (i.e., processed messages for handlers, success code for webhook sink) so that the pipeline knows to keep going.
Useful tips
Schema Changes: A change in the output schema of the external handler requires redeployment with schema_override.
- Failure Handling: The pipeline retries indefinitely with exponential backoff.
- Networking: Deploy the handler close to where the pipeline runs for better performance.
- Latency: Keep handler response times under 100ms to ensure smooth operation.
Was this page helpful?