SQS - Goldsky Docs

Overview

Send processed pipeline data to Amazon SQS queues for downstream processing, event-driven architectures, and decoupled integrations. Each row emitted by the upstream transform is serialized as a JSON object and delivered as the body of a single SQS message.

Configuration

Using Goldsky secrets (recommended)

sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    secret_name: MY_SQS_SECRET

Using direct credentials

Hardcoding credentials in pipeline definitions is not recommended for production use. Use secret_name with Goldsky secrets instead.

sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    access_key_id: <your-access-key>
    secret_access_key: <your-secret-key>
    region: us-east-1

Parameters

string

required

Must be sqs_sink

string

required

The transform or source to read data from

string

required

The full URL of your SQS queue (e.g., https://sqs.us-east-1.amazonaws.com/123456789012/my-queue)

string

Name of the secret containing SQS credentials. See Secret format below.

string

AWS access key ID for authentication. Not required if using secret_name.

string

AWS secret access key for authentication. Not required if using secret_name.

string

AWS region where the queue is located. Not required if using secret_name.

string

Optional custom SQS endpoint URL. Useful for pointing at SQS-compatible services or VPC endpoints.

string

Optional AWS session token for temporary credentials (e.g., STS, AWS SSO, assumed roles).

integer

default:"1000"

Runtime-level parameter — maximum number of rows the Turbo runtime buffers before delivering them to SQS. See Batching on the overview page. Independent of the SQS SendMessageBatch chunk size (10), which is applied on top by the sink.

string

default:"1s"

Runtime-level parameter — maximum time the Turbo runtime waits before flushing a batch, even if batch_size has not been reached. Accepts humantime durations such as 500ms, 1s, or 30s. See Batching.

Secret format

When using secret_name, create a Goldsky secret of type sqs with the following structure:

{
  "accessKeyId": "your-access-key-id",
  "secretAccessKey": "your-secret-access-key",
  "region": "us-east-1",
  "type": "sqs"
}

Create the secret using the Goldsky CLI:

goldsky secret create --name MY_SQS_SECRET --value '{
  "accessKeyId": "your-access-key-id",
  "secretAccessKey": "your-secret-access-key",
  "region": "us-east-1",
  "type": "sqs"
}'

IAM permissions

Your AWS credentials need the following IAM permissions on the target queue:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

Message format

Each row emitted by the upstream transform becomes the body of a single SQS message, serialized as a JSON object. For example, a row with columns id, from, and value is delivered as:

{"id": "0xabc...", "from": "0x123...", "value": "1000000000000000000"}

Delivery behavior

Batching: Messages are sent using the SQS SendMessageBatch API with the maximum supported chunk size of 10 messages per request. Larger upstream batches are split into multiple 10-message chunks automatically.
Retries: Partial batch failures are retried up to 5 times with exponential backoff (starting at 100ms, capped at 5s). Failures flagged as sender faults by SQS fail immediately without retry.
Queue type: Only standard SQS queues are supported. FIFO queues are not supported — the sink does not set MessageGroupId or MessageDeduplicationId, which FIFO queues require.

Example

Stream blockchain events to an SQS queue for downstream processing:

name: erc20-to-sqs
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  sqs_output:
    type: sqs_sink
    from: high_value
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/high-value-transfers
    secret_name: MY_SQS_SECRET

Best practices

Configure dead-letter queues

Set up a dead-letter queue (DLQ) in AWS to capture messages that fail processing. This helps with debugging and prevents data loss.

Monitor queue depth

Use CloudWatch to monitor your queue’s ApproximateNumberOfMessages metric. A growing backlog may indicate downstream processing issues.

Stay within the 256 KB message size limit

SQS rejects messages larger than 256 KB. Use an upstream SQL transform to drop or truncate large columns if your row payloads approach this limit.

​Overview

​Configuration

​Using Goldsky secrets (recommended)

​Using direct credentials

​Parameters

​Secret format

​IAM permissions

​Message format

​Delivery behavior

​Example

​Best practices

Overview

Configuration

Using Goldsky secrets (recommended)

Using direct credentials

Parameters

Secret format

IAM permissions

Message format

Delivery behavior

Example

Best practices