> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SQS

> Send data to Amazon Simple Queue Service (SQS)

## Overview

Send processed pipeline data to Amazon SQS queues for downstream processing, event-driven architectures, and decoupled integrations. Each row emitted by the upstream transform is serialized as a JSON object and delivered as the body of a single SQS message.

## Configuration

### Using Goldsky secrets (recommended)

```yaml theme={null}
sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    secret_name: MY_SQS_SECRET
```

### Using direct credentials

<Warning>
  Hardcoding credentials in pipeline definitions is not recommended for production use. Use `secret_name` with Goldsky secrets instead.
</Warning>

```yaml theme={null}
sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    access_key_id: <your-access-key>
    secret_access_key: <your-secret-key>
    region: us-east-1
```

## Parameters

<ParamField path="type" type="string" required>
  Must be `sqs_sink`
</ParamField>

<ParamField path="from" type="string" required>
  The transform or source to read data from
</ParamField>

<ParamField path="queue_url" type="string" required>
  The full URL of your SQS queue (e.g., `https://sqs.us-east-1.amazonaws.com/123456789012/my-queue`)
</ParamField>

<ParamField path="secret_name" type="string">
  Name of the secret containing SQS credentials. See [Secret format](#secret-format) below.
</ParamField>

<ParamField path="access_key_id" type="string">
  AWS access key ID for authentication. Not required if using `secret_name`.
</ParamField>

<ParamField path="secret_access_key" type="string">
  AWS secret access key for authentication. Not required if using `secret_name`.
</ParamField>

<ParamField path="region" type="string">
  AWS region where the queue is located. Not required if using `secret_name`.
</ParamField>

<ParamField path="endpoint_url" type="string">
  Optional custom SQS endpoint URL. Useful for pointing at SQS-compatible services or VPC endpoints.
</ParamField>

<ParamField path="session_token" type="string">
  Optional AWS session token for temporary credentials (e.g., STS, AWS SSO, assumed roles).
</ParamField>

## Secret format

When using `secret_name`, create a Goldsky secret of type `sqs` with the following structure:

```json theme={null}
{
  "accessKeyId": "your-access-key-id",
  "secretAccessKey": "your-secret-access-key",
  "region": "us-east-1",
  "type": "sqs"
}
```

Create the secret using the Goldsky CLI:

```bash theme={null}
goldsky secret create --name MY_SQS_SECRET --value '{
  "accessKeyId": "your-access-key-id",
  "secretAccessKey": "your-secret-access-key",
  "region": "us-east-1",
  "type": "sqs"
}'
```

## IAM permissions

Your AWS credentials need the following IAM permissions on the target queue:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}
```

## Message format

Each row emitted by the upstream transform becomes the body of a single SQS message, serialized as a JSON object. For example, a row with columns `id`, `from`, and `value` is delivered as:

```json theme={null}
{"id": "0xabc...", "from": "0x123...", "value": "1000000000000000000"}
```

## Delivery behavior

* **Batching**: Messages are sent using the SQS `SendMessageBatch` API with the maximum supported chunk size of 10 messages per request. Larger upstream batches are split into multiple 10-message chunks automatically.
* **Retries**: Partial batch failures are retried up to 5 times with exponential backoff (starting at 100ms, capped at 5s). Failures flagged as sender faults by SQS fail immediately without retry.
* **Queue type**: Only standard SQS queues are supported. FIFO queues are not supported — the sink does not set `MessageGroupId` or `MessageDeduplicationId`, which FIFO queues require.

## Example

Stream blockchain events to an SQS queue for downstream processing:

```yaml theme={null}
name: erc20-to-sqs
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  sqs_output:
    type: sqs_sink
    from: high_value
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/high-value-transfers
    secret_name: MY_SQS_SECRET
```

## Best practices

<AccordionGroup>
  <Accordion title="Configure dead-letter queues">
    Set up a dead-letter queue (DLQ) in AWS to capture messages that fail processing. This helps with debugging and prevents data loss.
  </Accordion>

  <Accordion title="Monitor queue depth">
    Use CloudWatch to monitor your queue's `ApproximateNumberOfMessages` metric. A growing backlog may indicate downstream processing issues.
  </Accordion>

  <Accordion title="Stay within the 256 KB message size limit">
    SQS rejects messages larger than 256 KB. Use an upstream SQL transform to drop or truncate large columns if your row payloads approach this limit.
  </Accordion>
</AccordionGroup>