Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Publish to an S2.dev stream. S2 is a serverless HTTP API for durable, append-only streams with support for high read fanout and bottomless storage on object storage. Use it to share streams of data with front-ends and back-ends as a simpler alternative to Kafka, AWS SQS, or Google Pub/Sub.
S2 is ideal for serverless architectures and teams using TypeScript or Python stacks who want a simpler alternative to Kafka for event streaming. Try the S2 Playground to get started.

​
Configuration

sinks:
  my_s2_sink:
    type: s2_sink
    from: my_transform
    access_token: your_access_token_here
    basin: your-basin-name
    stream: your-stream-name

​
Parameters

​
type
string
required
Must be s2_sink
​
from
string
required
The transform or source to read data from
​
access_token
string
required
S2 access token for authentication. Create one at s2.dev.
​
basin
string
required
Basin name. Must be a valid S2 BasinName.
​
stream
string
required
Stream name to append records to.

​
Behavior

  • Append-only: Each pipeline record is appended to the S2 stream as a single record. S2 streams are append-only by design, so updates and deletes are not supported.
  • JSON encoding: Records are serialized as line-delimited JSON (one JSON object per S2 record) before being appended.
  • Auto batching: Records are grouped into AppendRecordBatch instances up to S2’s per-batch size limits. When a batch fills up, the sink starts a new one and keeps appending.
  • Serverless: No infrastructure to manage. The sink talks to S2 over its HTTP API.

​
Example

Stream ERC-20 transfers from a Goldsky dataset to an S2 stream: 
name: erc20-transfers-to-s2
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  filtered_transfers:
    type: sql
    primary_key: id
    sql: |
      SELECT *
      FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  s2_output:
    type: s2_sink
    from: filtered_transfers
    access_token: your_access_token_here
    basin: your-basin-name
    stream: erc20-transfers