Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Publish processed pipeline data to Google Cloud Pub/Sub topics for fan-out to downstream consumers, event-driven architectures, and decoupled integrations. Each row emitted by the upstream transform is serialized as a JSON object and delivered as the body of a single Pub/Sub message.
The Pub/Sub sink is only available on Turbo pipelines.

​
Configuration

sinks:
  my_pubsub_sink:
    type: pubsub
    from: my_transform
    topic: my-topic
    secret_name: MY_PUBSUB_SECRET

​
Parameters

​
type
string
required
Must be pubsub.
​
from
string
required
The transform or source to read data from.
​
topic
string
required
The Pub/Sub topic to publish to. The topic must already exist in the GCP project referenced by the secret — the sink does not create topics. Specify the short topic id (for example my-topic), not the fully-qualified projects/.../topics/... name.
​
secret_name
string
required
Name of the Goldsky secret (type pubsub) containing the GCP project id and service-account credentials. See Secret format below.
​
batch_size
number
Maximum number of messages to batch before publishing.
​
batch_flush_interval
string
Maximum time to wait before flushing a batch. Accepts humantime durations such as 100ms, 1s, or 500ms.
​
description
string
Optional human-readable description for the sink.

​
Secret format

Create a Pub/Sub secret with the Goldsky CLI. You will be prompted for the GCP project id and the contents of a service-account JSON key file: 
goldsky secret create --name MY_PUBSUB_SECRET --type pubsub
The resulting secret has the following structure: 
{
  "type": "pubsub",
  "projectId": "your-gcp-project-id",
  "credentialsJson": "{\"type\":\"service_account\",\"project_id\":\"...\",\"private_key\":\"...\",\"client_email\":\"...\"}"
}
  • projectId: the GCP project that owns the destination topic (for example goldsky-prod).
  • credentialsJson: the raw contents of a GCP service-account JSON key file, on a single line. The CLI validates that the value parses as JSON and that type is service_account.

​
IAM permissions

The service account referenced by credentialsJson must have the following IAM roles in the destination project:
  • roles/pubsub.publisher — required to publish messages to the topic.
  • roles/pubsub.viewer — required for the sink’s topic-existence pre-check that runs during pipeline initialization.
You can also grant equivalent custom roles that include pubsub.topics.publish and pubsub.topics.get.

​
Message format

Each row emitted by the upstream transform becomes the data payload of a single Pub/Sub message, serialized as a JSON object. For example, a row with columns id, from, and value is delivered as: 
{"id": "0xabc...", "from": "0x123...", "value": "1000000000000000000"}

​
Example

Stream high-value ERC-20 transfers to a Pub/Sub topic for downstream fan-out: 
name: erc20-to-pubsub
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  pubsub_output:
    type: pubsub
    from: high_value
    topic: high-value-transfers
    secret_name: MY_PUBSUB_SECRET

​
Best practices

The sink performs a topic-existence check during initialization and will fail to start if the topic does not exist. Create the topic in advance with gcloud pubsub topics create my-topic or via Terraform.
Grant the service account roles/pubsub.publisher and roles/pubsub.viewer only on the specific topic (or the project that contains it) — not on the whole organization. Rotate the JSON key periodically.
Pub/Sub rejects messages larger than 10 MB. Use an upstream SQL transform to drop or truncate large columns if your row payloads approach this limit.
Pub/Sub fans out to subscriptions, not directly to subscribers. Create one subscription per downstream consumer so they receive independent copies of the stream.