> ## Documentation Index
> Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Troubleshooting Turbo pipelines
> Diagnose and fix common Turbo pipeline problems: error states, missing data, wrong output, sink write failures, and lag
When a pipeline misbehaves, the fastest path to a fix is almost always the same three moves: check the status, read the logs, watch the live data. This guide turns that into a repeatable diagnosis flow, walks through the most common symptoms end to end, and ends with a reference table of the error messages you are most likely to see in pipeline logs.
For project-wide metrics (lag, throughput, checkpoint health), see the [health dashboard](/turbo-pipelines/health-dashboard). For record-level debugging, see [Live Inspect](/turbo-pipelines/live-inspect).
## First checks
Run these in order before digging into a specific symptom. Each step rules out a class of problem.
```bash theme={"dark"}
goldsky project list
```
If this fails or reports that you are not logged in, generate an API key and run `goldsky login` before continuing. If `goldsky turbo list` reports that the turbo binary is not installed, install the [Turbo CLI extension](/turbo-pipelines/cli) first. A pipeline that "doesn't exist" is often just a pipeline in a different project.
```bash theme={"dark"}
goldsky turbo list
```
Every pipeline in the current project is listed with its current status. If your pipeline is missing, check the name spelling and the project you are logged into. See [pipeline states](#pipeline-states) below for what each status means.
```bash theme={"dark"}
goldsky turbo logs my-pipeline --tail 100
```
Healthy logs show steady progress lines; unhealthy logs show repeated `Execution error` lines or the same error recurring across restarts. See [reading logs](#reading-logs) below, and match any error text against the [error message reference](#error-message-reference).
```bash theme={"dark"}
goldsky turbo inspect my-pipeline
```
The [Live Inspect](/turbo-pipelines/live-inspect) TUI shows records as they move through each source, transform, and sink. A healthy pipeline shows records arriving on each node's tab. A TUI stuck on "Waiting for records..." means nothing is flowing. Jump to [no data is arriving](#no-data-is-arriving).
Once you know the status and have logs in hand, jump to the [walkthrough](#diagnosis-walkthroughs) that matches your symptom.
### Pipeline states
`goldsky turbo list` reports one of these statuses for each pipeline:
| Status | Meaning | What to do |
| ----------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `running` | Actively processing data | If something still looks wrong, the issue is data quality, latency, or configuration; see the walkthroughs below |
| `starting` | Initializing | Normal for the first few minutes. If it stays in `starting` for more than about 10 minutes, check the logs |
| `paused` | Manually paused (replicas set to 0) | Resume with `goldsky turbo resume my-pipeline` |
| `stopped` | Not running (manually stopped) | Redeploy with `goldsky turbo apply my-pipeline.yaml` |
| `error` | The pipeline hit a failure it could not recover from | Start with [pipeline is in an error state](#pipeline-is-in-an-error-state) |
| `completed` | A [job-mode](/turbo-pipelines/job-mode) pipeline finished its bounded range | Expected for jobs. The pipeline is auto-deleted about 1 hour after termination |
Job-mode pipelines (`job: true`) cannot be restarted. Delete the job and re-apply the YAML instead. They are auto-deleted about 1 hour after termination, whether they succeeded or failed, so capture logs before the cleanup window closes. See the [job mode](/turbo-pipelines/job-mode) guide.
### Reading logs
You do not need to read every line. Scan for these signals:
| Log line contains | What it tells you |
| -------------------- | ------------------------------------------------------------------------------------------------------------ |
| `Processing block` | The pipeline is actively processing data |
| `Checkpoint updated` | Progress is being durably saved |
| `rows written` | Data is reaching the sink |
| `Execution error` | Something failed. Match the rest of the line against the [error message reference](#error-message-reference) |
Useful variations when the last 100 lines are not enough:
```bash theme={"dark"}
# Stream logs in real time (streaming pipelines only, not jobs)
goldsky turbo logs my-pipeline -f
# Add timestamps to correlate with dashboard graphs
goldsky turbo logs my-pipeline --tail 200 --timestamps
# Only the last 30 minutes
goldsky turbo logs my-pipeline --since 1800
```
## Diagnosis walkthroughs
Each walkthrough uses `eth-transfers` as the example pipeline, a streaming pipeline reading `ethereum.erc20_transfers` and writing to a Postgres sink. Substitute your own pipeline name.
### Pipeline is in an error state
**Symptom:** `goldsky turbo list` shows the pipeline as `error`, or it crash-loops between `starting` and `error`.
**What to check:** the logs, which almost always name the failing component.
```bash theme={"dark"}
goldsky turbo logs eth-transfers --tail 100
```
**What you'll see:** an `Execution error` line identifying the root cause. For example, the single most common failure, bad database credentials in a secret:
```text theme={"dark"}
Execution error: Failed to create PostgreSQL connection: error returned from
database: password authentication failed for user 'app_user'
```
**Fix:** for this example, the secret holds a wrong username or password.
1. Verify the credentials work outside Goldsky, for example with `psql 'postgresql://user:pass@host:5432/db'`.
2. Recreate the secret with the corrected connection string. Secrets cannot be edited in place, so delete and create it again under the same name:
```bash theme={"dark"}
goldsky secret delete MY_POSTGRES
goldsky secret create MY_POSTGRES
```
3. Restart the pipeline so it picks up the new secret:
```bash theme={"dark"}
goldsky turbo restart eth-transfers
```
For any other error text, match it against the [error message reference](#error-message-reference) below. Connection, schema, resource, and transform failures each have a specific fix. If the logs show a connection or authentication error, also confirm every secret the pipeline references actually exists:
```bash theme={"dark"}
goldsky secret list
```
Cross-check each `secret_name` in your YAML against that list; a misspelled `secret_name` fails the same way as a missing secret.
### No data is arriving
**Symptom:** the pipeline is `running`, but the sink table is empty or stopped growing.
**What to check:** whether records are flowing at each hop (source, transform, sink) using Live Inspect in print mode. Start at the source:
```bash theme={"dark"}
goldsky turbo inspect eth-transfers -n eth_transfers -p
```
**What you'll see:** one of two things.
* **No records at all** ("Waiting for records..." with nothing printed). If the source uses `start_at: latest`, the pipeline only sees data created *after* deployment:
```yaml theme={"dark"}
sources:
eth_transfers:
type: dataset
dataset_name: ethereum.erc20_transfers
start_at: latest # Only new data from deploy time onward
```
Combined with a selective filter, it can legitimately take a long time for the first matching record to arrive. To process history instead, change `start_at` and redeploy. Because `apply` preserves checkpoints, you need `goldsky turbo restart eth-transfers --clear-state` (or a renamed source) to make the pipeline re-read from the new starting point.
* **Records at the source but nothing downstream.** Move node by node with `-n ` until records stop appearing:
* If records stop after a transform, the transform is filtering everything out. Check the WHERE clause and the exact spelling of the name in `FROM`: it must match the source key exactly.
* If records reach the last transform but never hit the sink, check the sink's `from:` field points at the node you think it does.
**Fix:** correct the YAML, run `goldsky turbo validate eth-transfers.yaml`, then `goldsky turbo apply eth-transfers.yaml`. Deploy with `-i` to reopen the inspect TUI and confirm records now reach the sink node.
### Output looks wrong
**Symptom:** data arrives, but rows are duplicated, fields are null or missing, or values don't match what the transform should produce.
**What to check:** compare a transform's input against its output. Open the inspect TUI and switch between node tabs, or capture samples of each node for a side-by-side look:
```bash theme={"dark"}
goldsky turbo inspect eth-transfers -n filtered_transfers -p | jq '.data' > transform-output.json
```
**What you'll see:** the actual records the transform emits, which usually makes the bug obvious:
* **Duplicate rows:** the `primary_key` column isn't unique per record, so upserts collide or multiply. Joins are a common source of accidental fan-out.
* **Missing or null fields:** the SQL doesn't select the column, or a [TypeScript transform](/turbo-pipelines/transforms/typescript) returns an object missing fields declared in its `schema`.
* **Wrong values:** check for type casts (numeric strings compared as text) and case-sensitive comparisons on address columns.
**Fix:** edit the transform, validate, and redeploy:
```bash theme={"dark"}
goldsky turbo validate eth-transfers.yaml
goldsky turbo apply eth-transfers.yaml
```
Because `apply` preserves checkpoints, the corrected logic only applies to new data. To rewrite history through the fixed transform, restart with cleared state:
```bash theme={"dark"}
goldsky turbo restart eth-transfers --clear-state
```
`--clear-state` discards all checkpoints and reprocesses from the beginning. Rows already written under the old logic are overwritten only where primary keys match; rows the fixed logic no longer produces will remain in the sink unless you clean them up yourself.
### Sink write failures
**Symptom:** logs show errors mentioning the sink, and lag starts growing. When a sink can't accept writes, backpressure deliberately slows the whole pipeline rather than dropping data.
**What to check:** the logs, which include the database's own error text:
```bash theme={"dark"}
goldsky turbo logs eth-transfers --tail 100
```
**What you'll see:** one of a handful of database-side failures. Examples:
```text theme={"dark"}
Execution error: Failed to create PostgreSQL connection: Connection refused
```
```text theme={"dark"}
Execution error: Failed to create table 'transfers': error returned from
database: could not extend file because project size limit (512 MB) has
been exceeded
```
**Fix:** depends on the message.
| Message | Fix |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Connection refused` | Verify host and port in the secret, confirm the database is up and accepts remote connections, and allow Goldsky's IPs through your firewall (see [static IPs](/turbo-pipelines/faq#does-turbo-support-static-ips)) |
| `SSL connection is required` | Managed providers (Neon, Supabase) handle SSL automatically; for self-hosted databases, enable SSL and include SSL parameters in the connection string |
| `project size limit ... exceeded` | The database is out of storage (Neon's free tier caps at 512 MB). Upgrade the plan, clear old data, or add filtering to reduce write volume |
| `duplicate key` / `primary key violation` | The transform feeding the sink emits non-unique primary keys; see [output looks wrong](#output-looks-wrong) |
After fixing the database side, the pipeline resumes on its own retries; if it sits in `error`, nudge it with `goldsky turbo restart eth-transfers`.
### Pipeline is stuck or lagging
**Symptom:** the pipeline is `running` and data arrives, but minutes or hours behind the chain tip, or throughput has visibly dropped.
**What to check:** the [health dashboard](/turbo-pipelines/health-dashboard) first. Its [where to look first](/turbo-pipelines/health-dashboard#where-to-look-first) flow separates source, pipeline, and sink bottlenecks. In short:
1. **Checkpoint failures non-zero?** Check the logs immediately: the pipeline is not durably saving progress.
2. **Block lag growing?** Look at sink flush latency next. Growing end-to-end lag is usually a slow sink applying backpressure, not a slow source.
3. **Sinks fast but checkpoint duration high?** Tune batching: raise `batch_size` or lower `batch_flush_interval` on the sink (see the [pipeline configuration reference](/turbo-pipelines/pipeline-config)).
Then confirm from the CLI side:
```bash theme={"dark"}
# Are error or backpressure warnings present?
goldsky turbo logs eth-transfers --tail 200 --timestamps
# Is data still moving at all?
goldsky turbo inspect eth-transfers -p
```
**What you'll see:**
* `out of memory` in the logs with periodic restarts: the pipeline is undersized. Raise `resource_size` (`s` → `m` → `l`) and redeploy.
* `backpressure` or lag warnings with high sink flush latency on the dashboard: the sink is the bottleneck. Add indexes for the upsert path, size up the database, or split load across sinks.
* Steadily processing but far behind after a fresh deploy: a pipeline backfilling history is *supposed* to show high lag while it catches up. Watch whether lag trends down; only intervene if it doesn't.
**Fix:** match the bottleneck: resources (`resource_size`), batching (`batch_size`, `batch_flush_interval`), or the sink database itself. For a transient wedge with no clear cause, a plain restart is safe and keeps checkpoints:
```bash theme={"dark"}
goldsky turbo restart eth-transfers
```
If you suspect checkpoint trouble, you can list the pipeline's checkpoint entries with `goldsky turbo state list eth-transfers` to see exactly what state exists (and what `--clear-state` would remove).
## Error message reference
Match text from `goldsky turbo logs` against these patterns. Grouped by category.
### Authentication
| Log message contains | What it means | Fix |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `password authentication failed for user` | The database credentials in the secret are wrong: bad password, bad username, or a user that doesn't exist | Verify the credentials work outside Goldsky (for example with `psql`), then delete and recreate the secret with the corrected connection string and restart the pipeline |
### Network
| Log message contains | What it means | Fix |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection refused` | The pipeline cannot reach the database server: it is down, the host or port in the secret is wrong, a firewall blocks Goldsky, or remote connections are disabled | Verify the database is running and the host/port are correct, allow Goldsky's IPs through the firewall ([static IPs](/turbo-pipelines/faq#does-turbo-support-static-ips)), and enable remote connections |
| `SSL connection is required` | The database requires SSL but the connection isn't using it | Managed databases (Neon, Supabase) handle SSL automatically; for self-hosted databases, enable SSL in the database configuration |
| `kafka ... timeout` or `broker ... not available` | The pipeline cannot connect to the Kafka broker | Verify the bootstrap servers, the security protocol (`SASL_SSL` vs. `PLAINTEXT`), and the Kafka credentials in the secret |
### Configuration
| Log message contains | What it means | Fix |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Secret '...' not found` | The pipeline references a secret that doesn't exist: never created, deleted, or misspelled in the YAML | Run `goldsky secret list`, check the `secret_name` spelling in the YAML, and create the missing secret with `goldsky secret create MY_SECRET` |
| `references unknown dataset` | The source names a dataset that doesn't exist, often a wrong chain prefix (`matic`, not `polygon`) or dataset type (`raw_transactions`, not `transactions`) | Check the dataset name against the [available datasets](https://app.goldsky.com/data-sources) and run `goldsky turbo validate pipeline.yaml` before deploying |
| `SQL syntax error` or `Parser error` | A SQL transform has invalid syntax or references a column that doesn't exist | Run `goldsky turbo validate pipeline.yaml`, check column names against the source schema, and test the query in a SQL client |
| `pipeline already exists` or `cannot update ... job` | Job-mode pipelines cannot be updated in place, and a previous run may not be cleaned up yet | Delete the existing job with `goldsky turbo delete my-pipeline`, then `goldsky turbo apply my-pipeline.yaml`, or wait for auto-cleanup about 1 hour after the job terminates |
### Storage
| Log message contains | What it means | Fix |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `project size limit ... has been exceeded` | The sink database's storage quota is full, common on Neon's free tier, which is capped at 512 MB | Upgrade the database plan, switch providers, clear existing data, or add filtering to the pipeline to reduce write volume |
### Data
| Log message contains | What it means | Fix |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `duplicate key` or `primary key violation` | The transform emits non-unique values in the `primary_key` column, often from a join that fans out rows | Ensure the `primary_key` column is unique per record, add `DISTINCT` to the query, or pick a different `primary_key` column |
| `checkpoint ... reset` or `starting from beginning` | The pipeline's checkpoint was reset and it is reprocessing from the start. Checkpoints are tied to the pipeline and source names, so renaming either resets them | Usually intentional (a rename or `--clear-state`). If not, restore the original pipeline and source names to resume from existing checkpoints |
### Resources
| Log message contains | What it means | Fix |
| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `out of memory`, `OOM`, or `memory limit exceeded` | The pipeline exceeded its memory allocation: too much data volume for its `resource_size`, or transforms accumulating large state | Increase `resource_size` (`s` → `m` → `l`), add filtering to reduce data volume, or split the workload into multiple pipelines |
### Performance
| Log message contains | What it means | Fix |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `processing ... slow`, `backpressure`, or `lag` | A warning, not a failure: the pipeline is processing slower than data arrives (undersized resources, a slow sink, or heavy transforms) | Increase `resource_size`, optimize the sink database (indexes on the upsert path), simplify transforms, or spread load across multiple sinks; see [pipeline is stuck or lagging](#pipeline-is-stuck-or-lagging) |
### Transforms
| Log message contains | What it means | Fix |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WASM execution failed`, `script transform error`, or `invoke is not a function` | A [TypeScript transform](/turbo-pipelines/transforms/typescript) failed at runtime: a null access, a missing `invoke(data)` function, a return value that doesn't match the declared `schema`, or use of unsupported APIs | Ensure the script defines `function invoke(data)`, add null checks (`input.field ?? ''`), return every field the `schema` declares, and remove `import`/`require` statements; the sandbox has no network or module access |
| `dynamic table ... error`, `lookup ... failed`, or `backend ... connection` | A [dynamic table](/turbo-pipelines/transforms/dynamic-tables) transform cannot connect to or query its backing store | Verify the secret exists (`goldsky secret list`), the backing table exists in the database, and column names match between the YAML and the actual table |
| `handler ... timeout`, `external ... handler ... failed`, or `HTTP ... timeout` | An [HTTP handler transform](/turbo-pipelines/transforms/http-handler)'s external endpoint is slow, unreachable, or returning a malformed response | Increase `timeout_ms`, verify the handler URL is reachable, make sure the endpoint returns a JSON array the same length as its input, and reduce `batch_size` to send fewer records per request |
## When the CLI itself misbehaves
Two failure modes live in the CLI rather than the pipeline:
* **Commands hang with no output.** The update notifier can stall on a failed network check. Disable it for one command to confirm:
```bash theme={"dark"}
GOLDSKY_NO_UPDATE_NOTIFIER=1 goldsky project list
```
If it still hangs, bound the command with a timeout and check basic connectivity to rule out a network problem:
```bash theme={"dark"}
timeout 30 goldsky project list
curl -I https://goldsky.com
```
If `goldsky` commands work but `turbo` subcommands hang, the turbo binary may be corrupted. Remove it and reinstall:
```bash theme={"dark"}
rm -f ~/.goldsky/bin/turbo
curl https://install-turbo.goldsky.com | sh
```
* **"The turbo binary is not installed."** Turbo is a separate CLI extension. Install (or reinstall) it, then verify with `goldsky turbo list`:
```bash theme={"dark"}
curl https://install-turbo.goldsky.com | sh
```
See the [Turbo CLI installation guide](/turbo-pipelines/cli) for details.
## Still stuck
If none of the above resolves it, [contact support](/getting-support). Including the following up front usually saves a round trip:
* The pipeline name and project.
* The pipeline definition: `goldsky turbo get my-pipeline`.
* Recent logs with timestamps: `goldsky turbo logs my-pipeline --tail 100 --timestamps`.
* The exact error messages you matched (or failed to match) in the reference above, and roughly when the problem started.
Job-mode pipelines are auto-deleted about 1 hour after termination. Capture the definition and logs before they disappear.
## Related topics
- [Turbo Pipelines (turbo)](/compose/context/turbo.md)
- [Deploy a Turbo pipeline](/turbo-pipelines/quickstart.md)
- [Turbo pipelines FAQ](/turbo-pipelines/faq.md)
- [Movement sources for Turbo pipelines](/turbo-pipelines/sources/movement.md)
- [Solana sources for Turbo pipelines](/turbo-pipelines/sources/solana.md)