> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Cloud Pub/Sub

> Publish processed data to Google Cloud Pub/Sub topics

## Overview

Publish processed pipeline data to [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) topics for fan-out to downstream consumers, event-driven architectures, and decoupled integrations. Each row emitted by the upstream transform is serialized as a JSON object and delivered as the body of a single Pub/Sub message.

<Note>
  The Pub/Sub sink is only available on Turbo pipelines.
</Note>

## Configuration

```yaml theme={null}
sinks:
  my_pubsub_sink:
    type: pubsub
    from: my_transform
    topic: my-topic
    secret_name: MY_PUBSUB_SECRET
```

## Parameters

<ParamField path="type" type="string" required>
  Must be `pubsub`.
</ParamField>

<ParamField path="from" type="string" required>
  The transform or source to read data from.
</ParamField>

<ParamField path="topic" type="string" required>
  The Pub/Sub topic to publish to. The topic must already exist in the GCP project referenced by the secret — the sink does not create topics. Specify the short topic id (for example `my-topic`), not the fully-qualified `projects/.../topics/...` name.
</ParamField>

<ParamField path="secret_name" type="string" required>
  Name of the Goldsky secret (type `pubsub`) containing the GCP project id and service-account credentials. See [Secret format](#secret-format) below.
</ParamField>

<ParamField path="batch_size" type="number">
  Maximum number of messages to batch before publishing.
</ParamField>

<ParamField path="batch_flush_interval" type="string">
  Maximum time to wait before flushing a batch. Accepts humantime durations such as `100ms`, `1s`, or `500ms`.
</ParamField>

<ParamField path="description" type="string">
  Optional human-readable description for the sink.
</ParamField>

## Secret format

Create a Pub/Sub secret with the Goldsky CLI. You will be prompted for the GCP project id and the contents of a service-account JSON key file:

```bash theme={null}
goldsky secret create --name MY_PUBSUB_SECRET --type pubsub
```

The resulting secret has the following structure:

```json theme={null}
{
  "type": "pubsub",
  "projectId": "your-gcp-project-id",
  "credentialsJson": "{\"type\":\"service_account\",\"project_id\":\"...\",\"private_key\":\"...\",\"client_email\":\"...\"}"
}
```

* `projectId`: the GCP project that owns the destination topic (for example `goldsky-prod`).
* `credentialsJson`: the raw contents of a GCP service-account JSON key file, on a single line. The CLI validates that the value parses as JSON and that `type` is `service_account`.

## IAM permissions

The service account referenced by `credentialsJson` must have the following IAM roles in the destination project:

* `roles/pubsub.publisher` — required to publish messages to the topic.
* `roles/pubsub.viewer` — required for the sink's topic-existence pre-check that runs during pipeline initialization.

You can also grant equivalent custom roles that include `pubsub.topics.publish` and `pubsub.topics.get`.

## Message format

Each row emitted by the upstream transform becomes the data payload of a single Pub/Sub message, serialized as a JSON object. For example, a row with columns `id`, `from`, and `value` is delivered as:

```json theme={null}
{"id": "0xabc...", "from": "0x123...", "value": "1000000000000000000"}
```

## Example

Stream high-value ERC-20 transfers to a Pub/Sub topic for downstream fan-out:

```yaml theme={null}
name: erc20-to-pubsub
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  pubsub_output:
    type: pubsub
    from: high_value
    topic: high-value-transfers
    secret_name: MY_PUBSUB_SECRET
```

## Best practices

<AccordionGroup>
  <Accordion title="Create the topic before starting the pipeline">
    The sink performs a topic-existence check during initialization and will fail to start if the topic does not exist. Create the topic in advance with `gcloud pubsub topics create my-topic` or via Terraform.
  </Accordion>

  <Accordion title="Scope service-account permissions tightly">
    Grant the service account `roles/pubsub.publisher` and `roles/pubsub.viewer` only on the specific topic (or the project that contains it) — not on the whole organization. Rotate the JSON key periodically.
  </Accordion>

  <Accordion title="Stay within the 10 MB message size limit">
    Pub/Sub rejects messages larger than 10 MB. Use an upstream SQL transform to drop or truncate large columns if your row payloads approach this limit.
  </Accordion>

  <Accordion title="Provision subscriptions for each consumer">
    Pub/Sub fans out to subscriptions, not directly to subscribers. Create one subscription per downstream consumer so they receive independent copies of the stream.
  </Accordion>
</AccordionGroup>