Overview Write data to S3-compatible object storage services.
Configuration Using Goldsky secrets (recommended) sinks : my_s3_sink : type : s3_sink from : my_transform secret_name : MY_S3_SECRET endpoint : https://s3.amazonaws.com bucket : my-bucket prefix : my-prefix # Optional
Using direct credentials Hardcoding credentials in pipeline definitions is not recommended for production use. Use secret_name with Goldsky secrets instead.
sinks : my_s3_sink : type : s3_sink from : my_transform bucket : my-bucket region : us-east-1 secret_name : MY_S3_SECRET prefix : my-prefix # Optional
The secret should contain your AWS credentials in the following format:
access_key_id:secret_access_key
For temporary credentials that include a session token:
access_key_id:secret_access_key:session_token
Create a secret using the Goldsky CLI:
goldsky secret create MY_S3_SECRET
The session_token is only required when using AWS temporary credentials from
Security Token Service (STS), such as when assuming an IAM role, using
federated access, or AWS SSO sessions.
Parameters The transform or source to read data from
S3-compatible endpoint URL (e.g., https://s3.amazonaws.com,
https://t3.storage.dev)
Access key ID for authentication. Not required if using secret_name.
Secret access key for authentication. Not required if using secret_name.
AWS region or auto for S3-compatible services. Not required if using secret_name.
Name of the secret containing S3 credentials. See Secret Format above. Optional path prefix for objects within the bucket
AWS session token for temporary credentials. Only needed when using STS temporary credentials (assumed roles, federated access, AWS SSO).
When using secret_name, create a Goldsky secret of type s3 with the following structure:
{ "accessKeyId" : "your-access-key-id" , "secretAccessKey" : "your-secret-access-key" , "region" : "us-east-1" }
Create the secret using the Goldsky CLI:
goldsky secret create MY_S3_SECRET --type s3 --value '{"accessKeyId": "...", "secretAccessKey": "...", "region": "us-east-1"}'
The S3 sink works with any S3-compatible storage service, including AWS S3,
MinIO, Cloudflare R2, and others. Set region: auto for non-AWS services.
Using plaintext credentials (not recommended)
You can specify credentials directly in the pipeline configuration, but this is discouraged for security reasons: sinks : my_s3_sink : type : s3_sink from : my_transform bucket : my-bucket region : us-east-1 access_key_id : <your-access-key> secret_access_key : <your-secret-key> session_token : <your-session-token> # Optional, for temporary credentials
Plaintext credentials in pipeline definitions are a security risk. Use secret_name instead.
