Operating pipelines

This guide covers the day-to-day operations of a Turbo pipeline: deploying, pausing, resuming, restarting, updating, monitoring, and deleting. Every operation is a goldsky turbo command; see the CLI reference for the complete list of flags.

Deploying a pipeline

There are two main ways to deploy a pipeline: in the web app or by using the CLI.

If you prefer to deploy pipelines using a web interface, check out Goldsky Flow, the visual canvas editor in the dashboard.

`apply` command + pipeline configuration

The goldsky turbo apply command expects a pipeline configuration file. For example:

base-logs.yaml

name: base-logs-pipeline
resource_size: s

sources:
  base_logs:
    type: dataset
    dataset_name: base.raw_logs
    version: 1.0.0
    start_at: latest

transforms: {}

sinks:
  postgres_base_logs:
    type: postgres
    from: base_logs
    schema: public
    table: base_logs
    secret_name: GOLDSKY_SECRET
    primary_key: id

Save the configuration in a file and run:

goldsky turbo apply base-logs.yaml

This validates the configuration, creates (or updates) the pipeline in your current project, and starts processing data. Pass -i to open live inspect immediately after deployment. You can also check a configuration without deploying it:

goldsky turbo validate base-logs.yaml

Most lifecycle commands accept either the pipeline name (base-logs-pipeline) or the YAML file via -f (-f base-logs.yaml); the name is read from the file’s name field.

Pausing a pipeline

goldsky turbo pause <name> temporarily stops a running pipeline while preserving its checkpoint state for later resumption:

goldsky turbo pause base-logs-pipeline

Because Turbo pipelines continuously checkpoint their progress, no snapshot step is needed before pausing; the pipeline will pick up where it left off when resumed.

Resuming a pipeline

Resume a previously paused pipeline with:

goldsky turbo resume base-logs-pipeline

The pipeline restores its running state and continues from the last checkpoint.

You can only resume a paused pipeline. Attempting to resume an already running pipeline will return an error.

Restarting a pipeline

goldsky turbo restart <name> triggers a fresh pod restart for a running or paused pipeline. This is useful for recovering from issues:

goldsky turbo restart base-logs-pipeline

To discard all checkpoints and reprocess data from the beginning:

goldsky turbo restart base-logs-pipeline --clear-state

Restart vs resume: use resume to restore a paused pipeline without restarting pods. Use restart when you need a fresh pod restart, such as after configuration changes or to recover from issues.

Restart is not supported for job-mode pipelines. Use delete and apply to recreate the job instead.

Applying updates to pipeline configuration

To update a pipeline, edit its YAML file and re-apply it. For example, to change the description and resource size:

base-logs.yaml

name: base-logs-pipeline
description: a new description for my pipeline
resource_size: xl
# ...sources, transforms, and sinks unchanged

goldsky turbo apply base-logs.yaml

When a pipeline with the same name already exists, apply performs a server-side upsert that preserves checkpoints: the pipeline continues from where it left off with the new configuration. This is the common flow when you found an issue with a transform or sink and want to fix it without reprocessing history.

To have a pipeline restart from scratch instead, rename the pipeline (or the specific source node) so the existing checkpoints aren’t reused, or run goldsky turbo restart <name> --clear-state.

For a complete reference on the configuration attributes you can apply, check the pipeline configuration reference.

Monitoring a pipeline

Turbo gives you three complementary views into a running pipeline:

Status: goldsky turbo list shows every pipeline in your project and its current state.
Logs: goldsky turbo logs <name> -f streams the pipeline’s runtime output in real time. See viewing logs for filtering options.
Live data: goldsky turbo inspect <name> opens an interactive TUI showing the actual records flowing through each source, transform, and sink. See the Live inspect guide.

For project-wide health signals (lag, throughput, checkpoint failures, and sink performance), use the health dashboard in the web app.

Inspecting checkpoint state

Turbo pipelines periodically checkpoint their position so they can resume after pauses, restarts, and updates. To see a pipeline’s checkpoint state entries:

goldsky turbo state list base-logs-pipeline

This is primarily useful when debugging checkpoint-related behavior or confirming what would be cleared by --clear-state.

Deleting a pipeline

Although paused pipelines don’t consume processing resources, it’s always nice to keep your project clean and remove pipelines you aren’t going to use any longer. You can delete pipelines with the goldsky turbo delete command:

goldsky turbo delete base-logs-pipeline

By default, deleting a pipeline also clears its checkpoint state. Pass --clear-state=false to retain the state, so re-applying a pipeline with the same name resumes from the last checkpoint.

Pipelines running in job mode don’t need manual cleanup; they are automatically deleted 1 hour after termination.

Turbo

Guides

Reference

Mirror (legacy)

Deploying a pipeline

`apply` command + pipeline configuration

Pausing a pipeline

Resuming a pipeline

Restarting a pipeline

Applying updates to pipeline configuration

Monitoring a pipeline

Inspecting checkpoint state

Deleting a pipeline

​Deploying a pipeline

​apply command + pipeline configuration

​Pausing a pipeline

​Resuming a pipeline

​Restarting a pipeline

​Applying updates to pipeline configuration

​Monitoring a pipeline

​Inspecting checkpoint state

​Deleting a pipeline

Deploying a pipeline

`apply` command + pipeline configuration

Pausing a pipeline

Resuming a pipeline

Restarting a pipeline

Applying updates to pipeline configuration

Monitoring a pipeline

Inspecting checkpoint state

Deleting a pipeline