S2 - Goldsky Docs

Overview

Publish to an S2.dev stream. S2 is a serverless HTTP API for durable, append-only streams with support for high read fanout and bottomless storage on object storage. Use it to share streams of data with front-ends and back-ends as a simpler alternative to Kafka, AWS SQS, or Google Pub/Sub.

S2 is ideal for serverless architectures and teams using TypeScript or Python stacks who want a simpler alternative to Kafka for event streaming. Try the S2 Playground to get started.

Configuration

sinks:
  my_s2_sink:
    type: s2_sink
    from: my_transform
    access_token: your_access_token_here
    basin: your-basin-name
    stream: your-stream-name

Parameters

string

required

Must be s2_sink

string

required

The transform or source to read data from

string

required

S2 access token for authentication. Create one at s2.dev.

string

required

Basin name. Must be a valid S2 BasinName.

string

required

Stream name to append records to.

integer

default:"1000"

Runtime-level parameter — maximum number of rows the Turbo runtime buffers before appending them to the S2 stream. See Batching on the overview page.

string

default:"1s"

Runtime-level parameter — maximum time the Turbo runtime waits before flushing a batch, even if batch_size has not been reached. Accepts humantime durations such as 500ms, 1s, or 30s. See Batching.

Behavior

Append-only: Each pipeline record is appended to the S2 stream as a single record. S2 streams are append-only by design, so updates and deletes are not supported.
JSON encoding: Records are serialized as line-delimited JSON (one JSON object per S2 record) before being appended.
Auto batching: Records are grouped into AppendRecordBatch instances up to S2’s per-batch size limits. When a batch fills up, the sink starts a new one and keeps appending.
Serverless: No infrastructure to manage. The sink talks to S2 over its HTTP API.

Example

Stream ERC-20 transfers from a Goldsky dataset to an S2 stream:

name: erc20-transfers-to-s2
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  filtered_transfers:
    type: sql
    primary_key: id
    sql: |
      SELECT *
      FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  s2_output:
    type: s2_sink
    from: filtered_transfers
    access_token: your_access_token_here
    basin: your-basin-name
    stream: erc20-transfers

​Overview

​Configuration

​Parameters

​Behavior

​Example

Overview

Configuration

Parameters

Behavior

Example