Skip to main content

​
Overview

Live Inspect allows you to monitor and debug your pipelines by viewing live data samples as it flows through each topology node (sources, transforms, and sinks). This is invaluable for verifying data transformations, debugging issues, and understanding how your pipeline processes data in real-time.
Use Live Inspect when debugging pipeline logic, verifying transform outputs, or understanding data flow through complex pipelines.

​
Basic Usage

Inspect live data from a running pipeline: 
goldsky turbo inspect my-pipeline
Or use a pipeline configuration file: 
goldsky turbo inspect my-pipeline.yaml
The command displays live data samples from each topology node in your pipeline, updating automatically at regular intervals.

​
Command Options

​
NAME_OR_FILE
string
required
Pipeline name or path to YAML configuration file
​
-n, --topology-node-keys
string
Comma-separated list of topology node keys to filter. Only shows data from the specified nodes. Example: -n source1,transform1
​
--config
string
Path to config file

​
Filtering Topology Nodes

By default, Live Inspect shows data from all nodes in your pipeline. Filter to specific nodes using the -n flag: 
# Inspect only a specific source
goldsky turbo inspect my-pipeline -n polygon_transfers

# Inspect multiple nodes
goldsky turbo inspect my-pipeline -n filtered_transfers,enriched_events

# Inspect only transforms
goldsky turbo inspect my-pipeline -n transform1,transform2,transform3
Use filtering to focus on specific parts of your pipeline when debugging complex data flows.

​
Sampling

Live Inspect uses intelligent sampling to display data from your pipeline without overwhelming the output. By default, it shows:
  • 15 records per topology node: A representative sample of the most recent records flowing through each node (source, transform, or sink)
  • 30-second refresh interval: Data updates every 30 seconds (only when data is flowing through the topology node) to provide real-time visibility while maintaining reasonable resource usage
The sampling behavior can be tuned to suit particular use cases, such as higher-frequency updates for debugging or different sample sizes for specific data volumes. If you need to adjust the sampling behavior for your use case, please reach out for support.

​
Example: Inspecting a Pipeline

Consider a pipeline that filters ERC-20 transfers: 
name: erc20-filtered-transfers

sources:
  polygon_erc20_transfers:
    type: dataset
    dataset_name: matic.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  filtered_transfers:
    type: sql
    primary_key: id
    sql: |
      SELECT *
      FROM polygon_erc20_transfers
      WHERE CAST(value AS DECIMAL) > 1000000

sinks:
  postgres_sink:
    type: postgres
    from: filtered_transfers
    schema: public
    table: erc20_transfers
    secret_name: MY_POSTGRES
To inspect this pipeline: 
goldsky turbo inspect erc20-filtered-transfers
You’ll see live data samples from:
  • polygon_erc20_transfers (source)
  • filtered_transfers (transform)
  • postgres_sink (sink)

​
Use Cases

Use Live Inspect to verify that your SQL transforms are filtering and transforming data correctly. Check the input and output of each transform to ensure your logic works as expected.
Quickly verify that your sources are producing data in the expected format and with the correct schema before it flows through transforms.

​
Troubleshooting

If you don’t see any data:
  • Verify your pipeline is running: goldsky turbo list
  • Check that data is flowing: goldsky turbo logs my-pipeline
  • Ensure you’re using the correct pipeline name

​
Next Steps

CLI Reference

Learn about other CLI commands

Pipeline Configuration

Understand pipeline YAML syntax

Transforms

Learn about data transformations

Quickstart

Get started with Turbo Pipelines