Skip to main content

​
Overview

Send processed pipeline data to Amazon SQS queues for downstream processing, event-driven architectures, and decoupled integrations.

​
Configuration

sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    secret_name: MY_SQS_SECRET
    batch_flush_interval: 1s

​
Using direct credentials

Hardcoding credentials in pipeline definitions is not recommended for production use. Use secret_name with Goldsky secrets instead.
sinks:
  my_sqs_sink:
    type: sqs_sink
    from: my_transform
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
    access_key_id: <your-access-key>
    secret_access_key: <your-secret-key>
    region: us-east-1
    batch_flush_interval: 1s

​
Parameters

​
type
string
required
Must be sqs_sink
​
from
string
required
The transform or source to read data from
​
queue_url
string
required
The full URL of your SQS queue (e.g., https://sqs.us-east-1.amazonaws.com/123456789012/my-queue)
​
secret_name
string
Name of the secret containing SQS credentials. See Secret format below.
​
access_key_id
string
AWS access key ID for authentication. Not required if using secret_name.
​
secret_access_key
string
AWS secret access key for authentication. Not required if using secret_name.
​
region
string
AWS region where the queue is located. Not required if using secret_name.
​
batch_flush_interval
string
How frequently to flush messages to SQS (e.g., 1s, 5s, 100ms). Lower values reduce latency but may increase costs.

​
Secret format

When using secret_name, create a Goldsky secret of type sqs with the following structure: 
{
  "accessKeyId": "your-access-key-id",
  "secretAccessKey": "your-secret-access-key",
  "region": "us-east-1"
}
Create the secret using the Goldsky CLI: 
goldsky secret create MY_SQS_SECRET --type sqs --value '{"accessKeyId": "...", "secretAccessKey": "...", "region": "us-east-1"}'

​
IAM permissions

Your AWS credentials need the following IAM permissions on the target queue: 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

​
Example

Stream blockchain events to an SQS queue for downstream processing: 
name: erc20-to-sqs
resource_size: s

sources:
  transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  high_value:
    type: sql
    primary_key: id
    sql: |
      SELECT * FROM transfers
      WHERE CAST(value AS DECIMAL) > 1000000000000000000

sinks:
  sqs_output:
    type: sqs_sink
    from: high_value
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/high-value-transfers
    secret_name: MY_SQS_SECRET
    batch_flush_interval: 1s

​
Best practices

  • Use shorter intervals (e.g., 100ms, 1s) for low-latency requirements
  • Use longer intervals (e.g., 5s, 10s) to reduce SQS API calls and costs
  • Balance between latency and cost based on your use case
Set up a dead-letter queue (DLQ) in AWS to capture messages that fail processing. This helps with debugging and prevents data loss.
Use CloudWatch to monitor your queue’s ApproximateNumberOfMessages metric. A growing backlog may indicate downstream processing issues.
If message ordering is critical, use a FIFO queue. Note that FIFO queues have lower throughput limits (300 messages/second without batching).