Google Cloud Pub/Sub

Overview

Publish processed pipeline data to Google Cloud Pub/Sub topics for fan-out to downstream consumers, event-driven architectures, and decoupled integrations. Each row emitted by the upstream transform is serialized as a JSON object and delivered as the body of a single Pub/Sub message.

The Pub/Sub sink is only available on Turbo pipelines.

Configuration

sinks:
  my_pubsub_sink:
    type: pubsub
    from: my_transform
    topic: my-topic
    secret_name: MY_PUBSUB_SECRET

Parameters

string

required

Must be pubsub.

string

required

The transform or source to read data from.

string

required

The Pub/Sub topic to publish to. The topic must already exist in the GCP project referenced by the secret — the sink does not create topics. Specify the short topic id (for example my-topic), not the fully-qualified projects/.../topics/... name.

string

required

Name of the Goldsky secret (type pubsub) containing the GCP project id and service-account credentials. See Secret format below.

number

Maximum number of messages to batch before publishing.

string

Maximum time to wait before flushing a batch. Accepts humantime durations such as 100ms, 1s, or 500ms.

string

Optional human-readable description for the sink.

Secret format

Create a Pub/Sub secret with the Goldsky CLI. You will be prompted for the GCP project id and the contents of a service-account JSON key file:

goldsky secret create --name MY_PUBSUB_SECRET --type pubsub

The resulting secret has the following structure:

{
  "type": "pubsub",
  "projectId": "your-gcp-project-id",
  "credentialsJson": "{\"type\":\"service_account\",\"project_id\":\"...\",\"private_key\":\"...\",\"client_email\":\"...\"}"
}

projectId: the GCP project that owns the destination topic (for example goldsky-prod).
credentialsJson: the raw contents of a GCP service-account JSON key file, on a single line. The CLI validates that the value parses as JSON and that type is service_account.

IAM permissions

The service account referenced by credentialsJson must have the following IAM roles in the destination project:

roles/pubsub.publisher — required to publish messages to the topic.
roles/pubsub.viewer — required for the sink’s topic-existence pre-check that runs during pipeline initialization.

You can also grant equivalent custom roles that include pubsub.topics.publish and pubsub.topics.get.

Message format

Each row emitted by the upstream transform becomes the data payload of a single Pub/Sub message, serialized as a JSON object. For example, a row with columns id, from, and value is delivered as:

{"id": "0xabc...", "from": "0x123...", "value": "1000000000000000000"}

Example

Stream high-value ERC-20 transfers to a Pub/Sub topic for downstream fan-out:

name: erc20-to-pubsub
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  pubsub_output:
    type: pubsub
    from: high_value
    topic: high-value-transfers
    secret_name: MY_PUBSUB_SECRET

Best practices

Create the topic before starting the pipeline

The sink performs a topic-existence check during initialization and will fail to start if the topic does not exist. Create the topic in advance with gcloud pubsub topics create my-topic or via Terraform.

Scope service-account permissions tightly

Grant the service account roles/pubsub.publisher and roles/pubsub.viewer only on the specific topic (or the project that contains it) — not on the whole organization. Rotate the JSON key periodically.

Stay within the 10 MB message size limit

Pub/Sub rejects messages larger than 10 MB. Use an upstream SQL transform to drop or truncate large columns if your row payloads approach this limit.

Provision subscriptions for each consumer

Pub/Sub fans out to subscriptions, not directly to subscribers. Create one subscription per downstream consumer so they receive independent copies of the stream.

Turbo

API

Reference

Overview

Configuration

Parameters

Secret format

IAM permissions

Message format

Example

Best practices

​Overview

​Configuration

​Parameters

​Secret format

​IAM permissions

​Message format

​Example

​Best practices

Overview

Configuration

Parameters

Secret format

IAM permissions

Message format

Example

Best practices