Troubleshooting Turbo pipelines

When a pipeline misbehaves, the fastest path to a fix is almost always the same three moves: check the status, read the logs, watch the live data. This guide turns that into a repeatable diagnosis flow, walks through the most common symptoms end to end, and ends with a reference table of the error messages you are most likely to see in pipeline logs. For project-wide metrics (lag, throughput, checkpoint health), see the health dashboard. For record-level debugging, see Live Inspect.

First checks

Run these in order before digging into a specific symptom. Each step rules out a class of problem.

Confirm the CLI is working and you are in the right project

goldsky project list

If this fails or reports that you are not logged in, generate an API key and run goldsky login before continuing. If goldsky turbo list reports that the turbo binary is not installed, install the Turbo CLI extension first. A pipeline that “doesn’t exist” is often just a pipeline in a different project.

Find the pipeline and check its status

goldsky turbo list

Every pipeline in the current project is listed with its current status. If your pipeline is missing, check the name spelling and the project you are logged into. See pipeline states below for what each status means.

Read recent logs

goldsky turbo logs my-pipeline --tail 100

Healthy logs show steady progress lines; unhealthy logs show repeated Execution error lines or the same error recurring across restarts. See reading logs below, and match any error text against the error message reference.

Watch live data flow through the pipeline

goldsky turbo inspect my-pipeline

The Live Inspect TUI shows records as they move through each source, transform, and sink. A healthy pipeline shows records arriving on each node’s tab. A TUI stuck on “Waiting for records…” means nothing is flowing. Jump to no data is arriving.

Once you know the status and have logs in hand, jump to the walkthrough that matches your symptom.

Pipeline states

goldsky turbo list reports one of these statuses for each pipeline:

Status	Meaning	What to do
`running`	Actively processing data	If something still looks wrong, the issue is data quality, latency, or configuration; see the walkthroughs below
`starting`	Initializing	Normal for the first few minutes. If it stays in `starting` for more than about 10 minutes, check the logs
`paused`	Manually paused (replicas set to 0)	Resume with `goldsky turbo resume my-pipeline`
`stopped`	Not running (manually stopped)	Redeploy with `goldsky turbo apply my-pipeline.yaml`
`error`	The pipeline hit a failure it could not recover from	Start with pipeline is in an error state
`completed`	A job-mode pipeline finished its bounded range	Expected for jobs. The pipeline is auto-deleted about 1 hour after termination

Job-mode pipelines (job: true) cannot be restarted. Delete the job and re-apply the YAML instead. They are auto-deleted about 1 hour after termination, whether they succeeded or failed, so capture logs before the cleanup window closes. See the job mode guide.

Reading logs

You do not need to read every line. Scan for these signals:

Log line contains	What it tells you
`Processing block`	The pipeline is actively processing data
`Checkpoint updated`	Progress is being durably saved
`rows written`	Data is reaching the sink
`Execution error`	Something failed. Match the rest of the line against the error message reference

Useful variations when the last 100 lines are not enough:

# Stream logs in real time (streaming pipelines only, not jobs)
goldsky turbo logs my-pipeline -f

# Add timestamps to correlate with dashboard graphs
goldsky turbo logs my-pipeline --tail 200 --timestamps

# Only the last 30 minutes
goldsky turbo logs my-pipeline --since 1800

Diagnosis walkthroughs

Each walkthrough uses eth-transfers as the example pipeline, a streaming pipeline reading ethereum.erc20_transfers and writing to a Postgres sink. Substitute your own pipeline name.

Pipeline is in an error state

Symptom: goldsky turbo list shows the pipeline as error, or it crash-loops between starting and error. What to check: the logs, which almost always name the failing component.

goldsky turbo logs eth-transfers --tail 100

What you’ll see: an Execution error line identifying the root cause. For example, the single most common failure, bad database credentials in a secret:

Execution error: Failed to create PostgreSQL connection: error returned from
database: password authentication failed for user 'app_user'

Fix: for this example, the secret holds a wrong username or password.

Verify the credentials work outside Goldsky, for example with psql 'postgresql://user:pass@host:5432/db'.
Recreate the secret with the corrected connection string. Secrets cannot be edited in place, so delete and create it again under the same name:
goldsky secret delete MY_POSTGRES goldsky secret create MY_POSTGRES
Restart the pipeline so it picks up the new secret:
goldsky turbo restart eth-transfers

For any other error text, match it against the error message reference below. Connection, schema, resource, and transform failures each have a specific fix. If the logs show a connection or authentication error, also confirm every secret the pipeline references actually exists:

goldsky secret list

Cross-check each secret_name in your YAML against that list; a misspelled secret_name fails the same way as a missing secret.

No data is arriving

Symptom: the pipeline is running, but the sink table is empty or stopped growing. What to check: whether records are flowing at each hop (source, transform, sink) using Live Inspect in print mode. Start at the source:

goldsky turbo inspect eth-transfers -n eth_transfers -p

What you’ll see: one of two things.

No records at all (“Waiting for records…” with nothing printed). If the source uses start_at: latest, the pipeline only sees data created after deployment:
sources: eth_transfers: type: dataset dataset_name: ethereum.erc20_transfers start_at: latest # Only new data from deploy time onward
Combined with a selective filter, it can legitimately take a long time for the first matching record to arrive. To process history instead, change start_at and redeploy. Because apply preserves checkpoints, you need goldsky turbo restart eth-transfers --clear-state (or a renamed source) to make the pipeline re-read from the new starting point.
Records at the source but nothing downstream. Move node by node with -n <node-name> until records stop appearing:
- If records stop after a transform, the transform is filtering everything out. Check the WHERE clause and the exact spelling of the name in FROM: it must match the source key exactly.
- If records reach the last transform but never hit the sink, check the sink’s from: field points at the node you think it does.

Fix: correct the YAML, run goldsky turbo validate eth-transfers.yaml, then goldsky turbo apply eth-transfers.yaml. Deploy with -i to reopen the inspect TUI and confirm records now reach the sink node.

Output looks wrong

Symptom: data arrives, but rows are duplicated, fields are null or missing, or values don’t match what the transform should produce. What to check: compare a transform’s input against its output. Open the inspect TUI and switch between node tabs, or capture samples of each node for a side-by-side look:

goldsky turbo inspect eth-transfers -n filtered_transfers -p | jq '.data' > transform-output.json

What you’ll see: the actual records the transform emits, which usually makes the bug obvious:

Duplicate rows: the primary_key column isn’t unique per record, so upserts collide or multiply. Joins are a common source of accidental fan-out.
Missing or null fields: the SQL doesn’t select the column, or a TypeScript transform returns an object missing fields declared in its schema.
Wrong values: check for type casts (numeric strings compared as text) and case-sensitive comparisons on address columns.

Fix: edit the transform, validate, and redeploy:

goldsky turbo validate eth-transfers.yaml
goldsky turbo apply eth-transfers.yaml

Because apply preserves checkpoints, the corrected logic only applies to new data. To rewrite history through the fixed transform, restart with cleared state:

goldsky turbo restart eth-transfers --clear-state

--clear-state discards all checkpoints and reprocesses from the beginning. Rows already written under the old logic are overwritten only where primary keys match; rows the fixed logic no longer produces will remain in the sink unless you clean them up yourself.

Sink write failures

Symptom: logs show errors mentioning the sink, and lag starts growing. When a sink can’t accept writes, backpressure deliberately slows the whole pipeline rather than dropping data. What to check: the logs, which include the database’s own error text:

goldsky turbo logs eth-transfers --tail 100

What you’ll see: one of a handful of database-side failures. Examples:

Execution error: Failed to create PostgreSQL connection: Connection refused

Execution error: Failed to create table 'transfers': error returned from
database: could not extend file because project size limit (512 MB) has
been exceeded

Fix: depends on the message.

Message	Fix
`Connection refused`	Verify host and port in the secret, confirm the database is up and accepts remote connections, and allow Goldsky’s IPs through your firewall (see static IPs)
`SSL connection is required`	Managed providers (Neon, Supabase) handle SSL automatically; for self-hosted databases, enable SSL and include SSL parameters in the connection string
`project size limit ... exceeded`	The database is out of storage (Neon’s free tier caps at 512 MB). Upgrade the plan, clear old data, or add filtering to reduce write volume
`duplicate key` / `primary key violation`	The transform feeding the sink emits non-unique primary keys; see output looks wrong

After fixing the database side, the pipeline resumes on its own retries; if it sits in error, nudge it with goldsky turbo restart eth-transfers.

Pipeline is stuck or lagging

Symptom: the pipeline is running and data arrives, but minutes or hours behind the chain tip, or throughput has visibly dropped. What to check: the health dashboard first. Its where to look first flow separates source, pipeline, and sink bottlenecks. In short:

Checkpoint failures non-zero? Check the logs immediately: the pipeline is not durably saving progress.
Block lag growing? Look at sink flush latency next. Growing end-to-end lag is usually a slow sink applying backpressure, not a slow source.
Sinks fast but checkpoint duration high? Tune batching: raise batch_size or lower batch_flush_interval on the sink (see the pipeline configuration reference).

Then confirm from the CLI side:

# Are error or backpressure warnings present?
goldsky turbo logs eth-transfers --tail 200 --timestamps

# Is data still moving at all?
goldsky turbo inspect eth-transfers -p

What you’ll see:

out of memory in the logs with periodic restarts: the pipeline is undersized. Raise resource_size (s → m → l) and redeploy.
backpressure or lag warnings with high sink flush latency on the dashboard: the sink is the bottleneck. Add indexes for the upsert path, size up the database, or split load across sinks.
Steadily processing but far behind after a fresh deploy: a pipeline backfilling history is supposed to show high lag while it catches up. Watch whether lag trends down; only intervene if it doesn’t.

Fix: match the bottleneck: resources (resource_size), batching (batch_size, batch_flush_interval), or the sink database itself. For a transient wedge with no clear cause, a plain restart is safe and keeps checkpoints:

goldsky turbo restart eth-transfers

If you suspect checkpoint trouble, you can list the pipeline’s checkpoint entries with goldsky turbo state list eth-transfers to see exactly what state exists (and what --clear-state would remove).

Error message reference

Match text from goldsky turbo logs against these patterns. Grouped by category.

Authentication

Log message contains	What it means	Fix
`password authentication failed for user`	The database credentials in the secret are wrong: bad password, bad username, or a user that doesn’t exist	Verify the credentials work outside Goldsky (for example with `psql`), then delete and recreate the secret with the corrected connection string and restart the pipeline

Network

Log message contains	What it means	Fix
`connection refused`	The pipeline cannot reach the database server: it is down, the host or port in the secret is wrong, a firewall blocks Goldsky, or remote connections are disabled	Verify the database is running and the host/port are correct, allow Goldsky’s IPs through the firewall (static IPs), and enable remote connections
`SSL connection is required`	The database requires SSL but the connection isn’t using it	Managed databases (Neon, Supabase) handle SSL automatically; for self-hosted databases, enable SSL in the database configuration
`kafka ... timeout` or `broker ... not available`	The pipeline cannot connect to the Kafka broker	Verify the bootstrap servers, the security protocol (`SASL_SSL` vs. `PLAINTEXT`), and the Kafka credentials in the secret

Configuration

Log message contains	What it means	Fix
`Secret '...' not found`	The pipeline references a secret that doesn’t exist: never created, deleted, or misspelled in the YAML	Run `goldsky secret list`, check the `secret_name` spelling in the YAML, and create the missing secret with `goldsky secret create MY_SECRET`
`references unknown dataset`	The source names a dataset that doesn’t exist, often a wrong chain prefix (`matic`, not `polygon`) or dataset type (`raw_transactions`, not `transactions`)	Check the dataset name against the available datasets and run `goldsky turbo validate pipeline.yaml` before deploying
`SQL syntax error` or `Parser error`	A SQL transform has invalid syntax or references a column that doesn’t exist	Run `goldsky turbo validate pipeline.yaml`, check column names against the source schema, and test the query in a SQL client
`pipeline already exists` or `cannot update ... job`	Job-mode pipelines cannot be updated in place, and a previous run may not be cleaned up yet	Delete the existing job with `goldsky turbo delete my-pipeline`, then `goldsky turbo apply my-pipeline.yaml`, or wait for auto-cleanup about 1 hour after the job terminates

Storage

Log message contains	What it means	Fix
`project size limit ... has been exceeded`	The sink database’s storage quota is full, common on Neon’s free tier, which is capped at 512 MB	Upgrade the database plan, switch providers, clear existing data, or add filtering to the pipeline to reduce write volume

Data

Log message contains	What it means	Fix
`duplicate key` or `primary key violation`	The transform emits non-unique values in the `primary_key` column, often from a join that fans out rows	Ensure the `primary_key` column is unique per record, add `DISTINCT` to the query, or pick a different `primary_key` column
`checkpoint ... reset` or `starting from beginning`	The pipeline’s checkpoint was reset and it is reprocessing from the start. Checkpoints are tied to the pipeline and source names, so renaming either resets them	Usually intentional (a rename or `--clear-state`). If not, restore the original pipeline and source names to resume from existing checkpoints

Resources

Log message contains	What it means	Fix
`out of memory`, `OOM`, or `memory limit exceeded`	The pipeline exceeded its memory allocation: too much data volume for its `resource_size`, or transforms accumulating large state	Increase `resource_size` (`s` → `m` → `l`), add filtering to reduce data volume, or split the workload into multiple pipelines

Performance

Log message contains	What it means	Fix
`processing ... slow`, `backpressure`, or `lag`	A warning, not a failure: the pipeline is processing slower than data arrives (undersized resources, a slow sink, or heavy transforms)	Increase `resource_size`, optimize the sink database (indexes on the upsert path), simplify transforms, or spread load across multiple sinks; see pipeline is stuck or lagging

Transforms

Log message contains	What it means	Fix
`WASM execution failed`, `script transform error`, or `invoke is not a function`	A TypeScript transform failed at runtime: a null access, a missing `invoke(data)` function, a return value that doesn’t match the declared `schema`, or use of unsupported APIs	Ensure the script defines `function invoke(data)`, add null checks (`input.field ?? ''`), return every field the `schema` declares, and remove `import`/`require` statements; the sandbox has no network or module access
`dynamic table ... error`, `lookup ... failed`, or `backend ... connection`	A dynamic table transform cannot connect to or query its backing store	Verify the secret exists (`goldsky secret list`), the backing table exists in the database, and column names match between the YAML and the actual table
`handler ... timeout`, `external ... handler ... failed`, or `HTTP ... timeout`	An HTTP handler transform’s external endpoint is slow, unreachable, or returning a malformed response	Increase `timeout_ms`, verify the handler URL is reachable, make sure the endpoint returns a JSON array the same length as its input, and reduce `batch_size` to send fewer records per request

When the CLI itself misbehaves

Two failure modes live in the CLI rather than the pipeline:

Commands hang with no output. The update notifier can stall on a failed network check. Disable it for one command to confirm:
GOLDSKY_NO_UPDATE_NOTIFIER=1 goldsky project list
If it still hangs, bound the command with a timeout and check basic connectivity to rule out a network problem:
timeout 30 goldsky project list curl -I https://goldsky.com
If goldsky commands work but turbo subcommands hang, the turbo binary may be corrupted. Remove it and reinstall:
rm -f ~/.goldsky/bin/turbo curl https://install-turbo.goldsky.com | sh
“The turbo binary is not installed.” Turbo is a separate CLI extension. Install (or reinstall) it, then verify with goldsky turbo list:
curl https://install-turbo.goldsky.com | sh

See the Turbo CLI installation guide for details.

Still stuck

If none of the above resolves it, contact support. Including the following up front usually saves a round trip:

The pipeline name and project.
The pipeline definition: goldsky turbo get my-pipeline.
Recent logs with timestamps: goldsky turbo logs my-pipeline --tail 100 --timestamps.
The exact error messages you matched (or failed to match) in the reference above, and roughly when the problem started.

Job-mode pipelines are auto-deleted about 1 hour after termination. Capture the definition and logs before they disappear.

Turbo

Guides

Reference

Mirror (legacy)

Troubleshooting Turbo pipelines

First checks

Pipeline states

Reading logs

Diagnosis walkthroughs

Pipeline is in an error state

No data is arriving

Output looks wrong

Sink write failures

Pipeline is stuck or lagging

Error message reference

Authentication

Network

Configuration

Storage

Data

Resources

Performance

Transforms

When the CLI itself misbehaves

Still stuck

​First checks

​Pipeline states

​Reading logs

​Diagnosis walkthroughs

​Pipeline is in an error state

​No data is arriving

​Output looks wrong

​Sink write failures

​Pipeline is stuck or lagging

​Error message reference

​Authentication

​Network

​Configuration

​Storage

​Data

​Resources

​Performance

​Transforms

​When the CLI itself misbehaves

​Still stuck

First checks

Pipeline states

Reading logs

Diagnosis walkthroughs

Pipeline is in an error state

No data is arriving

Output looks wrong

Sink write failures

Pipeline is stuck or lagging

Error message reference

Authentication

Network

Configuration

Storage

Data

Resources

Performance

Transforms

When the CLI itself misbehaves

Still stuck