Secrets

Overview

Secrets store the credentials Goldsky needs to connect to your infrastructure: database connection details, message queue credentials, cloud provider keys, and HTTP auth headers. They are stored securely in your Goldsky project and referenced by name from your pipeline configuration, so credentials never appear in your YAML.

Managing secrets

Create and manage secrets with the goldsky secret command:

# Create a secret (the CLI prompts for the type and value)
goldsky secret create MY_POSTGRES_SECRET

# List existing secrets
goldsky secret list

# Show a secret's stored value
goldsky secret reveal MY_POSTGRES_SECRET

# Delete a secret (add -f to skip the confirmation prompt in scripts)
goldsky secret delete MY_POSTGRES_SECRET

Run goldsky secret -h for the full list of commands.

Deleting a secret that a running pipeline references causes that pipeline to fail on its next connection. Check for references before deleting.

Secret types and the interactive prompt

Running goldsky secret create with no --value starts an interactive prompt that first asks you to pick a secret type, then collects the fields that type requires (host, credentials, region, etc.) and prints the permissions the credentials need before you paste them in. Every stored secret is tagged with one of these types:

Type	Used for
`jdbc`	Postgres and MySQL sinks. Enter a connection string or type in host/port/user/pass etc.
`clickHouse`	ClickHouse sinks. URL, username, password, database name
`elasticSearch`	Elasticsearch sinks (Mirror legacy). Host URL, username, password
`opensearch`	OpenSearch sinks. Host URL, username, password
`kafka`	Kafka sinks, with `PLAINTEXT`, `SASL_PLAINTEXT`, or `SASL_SSL` protocols
`s3`	S3 and S3-compatible object storage sinks. Access key, secret key, region
`sqs`	AWS SQS sinks. Access key, secret key, region
`pubsub`	Google Cloud Pub/Sub sinks. GCP project ID plus a service-account JSON key
`dynamodb`	AWS DynamoDB sinks. Access key, secret key, region
`httpauth`	Webhook sinks. Stores a single auth header as a `secretKey` / `secretValue` pair

Compose secrets are created through the goldsky compose CLI.

Turbo webhook sinks accept an httpauth secret via secret_name, but the header it injects cannot also appear in the sink’s inline headers: field. Pick one or the other. See the Turbo webhook sink for details.

Naming secrets

Secret names can only contain alphanumeric characters, underscores (_), and hyphens (-). Use descriptive uppercase names that encode environment and purpose (PROD_POSTGRES_MAIN, STAGING_CLICKHOUSE) rather than secret1 or postgres, so a goldsky secret list stays legible as your project grows.

Rotating credentials

Update a secret in place to rotate credentials without touching any pipeline YAML:

goldsky secret update MY_POSTGRES_SECRET --value 'postgres://admin:NEW_PASSWORD@db.example.com:5432/mydb'

Running pipelines pick up the new value on their next connection.

Referencing a secret in a pipeline

Reference a secret by name in the secret_name field of a sink or transform:

sinks:
  postgres_output:
    type: postgres
    from: filtered_transfers
    schema: public
    table: erc20_transfers
    secret_name: MY_POSTGRES_SECRET # References the stored secret
    primary_key: id

Secret formats

Each sink or transform type expects its own secret format. For Turbo pipelines, database secrets are created from a connection string; queue and storage secrets are JSON objects; HTTP secrets store a header name and value.

Used by	Format	Details
PostgreSQL sink, dynamic tables	`postgres://user:password@host:port/database`	Postgres sink
ClickHouse sink	`https://user:password@host:port/database` (`clickHouse` type)	ClickHouse sink § Secret format
MySQL sink	`mysql://user:password@host:3306/database` or individual fields	MySQL sink § Secret format
Kafka sink	JSON object (`bootstrapServers`, SASL credentials, schema registry)	Kafka sink § Secret structure
SQS sink	JSON object (`accessKeyId`, `secretAccessKey`, `region`)	SQS sink § Secret format
S3 sink	JSON object (`accessKeyId`, `secretAccessKey`, `region`)	S3 sink § Secret format
Pub/Sub sink	`pubsub` type (GCP project id + service account key)	Pub/Sub sink § Secret format
Webhook sinks, HTTP handler transforms	`httpauth` type (header name + value)	Webhook sink § Secret creation

Example: PostgreSQL

Postgres is the most common case. Create the secret and paste a standard connection string when prompted:

goldsky secret create MY_POSTGRES_SECRET

postgres://goldsky_writer:your_secure_password@db.example.com:5432/your_database?sslmode=require

See the Postgres sink page for database role setup, provider-specific notes, and the inline JSON alternative.

If you use Goldsky-hosted Postgres, provisioning the database automatically registers a secret in your project. There is no separate goldsky secret create step.

Mirror (legacy) secrets

Mirror pipelines use a different format for database secrets: a JSON object with individual connection fields instead of a connection string. For example, a Mirror Postgres secret looks like:

goldsky secret create --name MY_POSTGRES_SECRET --value '{
  "type": "jdbc",
  "protocol": "postgresql",
  "host": "db.host.com",
  "port": 5432,
  "databaseName": "myDatabase",
  "user": "myUser",
  "password": "myPassword"
}'

When migrating a Mirror pipeline to Turbo, recreate database secrets in the Turbo connection-string format; see the migration guide. Formats for Mirror-only sinks are documented on the surviving legacy pages, for example Elasticsearch. Mirror pipelines: the guided goldsky pipeline create <pipeline-name> flow lists your existing secrets and offers to create a new one as part of pipeline creation, so you don’t need to create a secret beforehand.

Troubleshooting

Symptom	Cause and fix
`Secret 'MY_SECRET' not found`	The name doesn’t exist or is misspelled. Run `goldsky secret list` and check the exact name.
`Secret 'MY_SECRET' already exists`	The name is taken. Use `goldsky secret update MY_SECRET --value "..."` to change its value, or pick a different name.
`Invalid JSON in secret value`	Syntax error in a JSON-typed secret. Validate first with `echo '{...}' \| jq .` before creating.
Pipeline fails with `connection refused`	Credentials are wrong or the database is unreachable. Verify outside Goldsky (`psql "postgresql://..."`), check the stored value with `goldsky secret reveal`, and make sure your database accepts connections from Goldsky’s IPs.
Pipeline fails with `authentication failed`	Wrong username or password. Update the secret with corrected credentials.
Password contains special characters	In connection strings, URL-encode special characters. In JSON-typed secrets the password is its own field, so most characters work as-is; escape backslashes (`\\`), quotes (`\"`), and newlines (`\n`).

Get started

Solutions

Account & billing

Security & operations

Resources

Overview

Managing secrets

Secret types and the interactive prompt

Naming secrets

Rotating credentials

Referencing a secret in a pipeline

Secret formats

Example: PostgreSQL

Mirror (legacy) secrets

Troubleshooting

​Overview

​Managing secrets

​Secret types and the interactive prompt

​Naming secrets

​Rotating credentials

​Referencing a secret in a pipeline

​Secret formats

​Example: PostgreSQL

​Mirror (legacy) secrets

​Troubleshooting

Overview

Managing secrets

Secret types and the interactive prompt

Naming secrets

Rotating credentials

Referencing a secret in a pipeline

Secret formats

Example: PostgreSQL

Mirror (legacy) secrets

Troubleshooting