​
What are channels?

Channels are a special type of Sink that represent intermediate storage layers designed to absorb the Goldsky firehose. They aren’t designed to be queryable on their own - instead, you should plan to connect them to your existing data stack or sink that’s not currently supported by Goldsky.

AWS S3

AWS S3 offers unparalleled scalability and durability for storing vast amounts of data.

AWS SQS

AWS SQS provides reliable message queuing for decoupling distributed systems with ease.

Kafka

Kafka excels in handling high-throughput, real-time data streams with strong fault tolerance.

​
What should I use?

​
For durable storage

Goldsky supports export of raw blockchain or custom data to AWS S3 or GCS, either in Iceberg format or in plain Parquet format. GCS support is currently available upon request, please reach out to us at support@goldsky.com. Keep in mind that the blockchain is eventually consistent, so reorgs in append-only mode is portrayed differently. Once data is in S3, you can use a solution like AWS Athena to query, or merge with existing spark pipelines.

​
For processing events as they come

If your backend needs to process and receive data as it appears on the blockchain, you can consider using our SQS or Kafka channel sinks. These sinks are append-only, so chain reorgs are not handled for you, but you do receive all the metadata required to handle reorgs yourself. An example architecture would have:
  1. An SQS queue
  2. An AWS Lambda function processing said queue
Mirror will send each event as a message into the SQS queue, and the lambda function can process it however you need through making additional enrichments, calling discord/telegram bots, or inserting into another datastore that we don’t yet support.

​
For more processing

In addition to AWS S3, we support direct emits to Kafka. Kafka can store messages at scale, making it a great choice as an initial holding place for data before you do further processing. Our team can work with many different strategies and can give guidance on how to integrate with our data format inside Kafka. Reach out to our support team at support@goldsky.com if you’d like to learn more.