Turbo pipeline advanced metrics dashboard reference

Overview

The Advanced metrics dashboard is a Grafana dashboard that shows detailed metrics for a single Turbo pipeline. It goes deeper than the project-wide Health overview: it breaks metrics down per component (each source, transform, and sink), and it adds checkpoint internals and container resource usage. To open it:

Sign in to the Goldsky dashboard and open a pipeline from the Pipelines page.
Click Advanced metrics. Grafana opens in a new tab with the dashboard filtered to that pipeline.

You can also open it by clicking a pipeline name in the Pipeline Status panel of the Health overview.

Use the Health overview to find the pipeline that has a problem. Then use this dashboard to find the component that causes it.

Filters

The filters at the top of the dashboard apply to all panels.

Filter	What it does
Service Instance ID	The pipeline to show. Set automatically when you open the dashboard from a pipeline.
Component ID	Limit panels to one or more components, by the name you gave them in the pipeline YAML.
Node Type	Limit panels to `source`, `transform`, or `sink` components.
Operator Type	Limit panels to one implementation, for example `sql`, `postgres`, or `webhook`.
Rate Time Window	The time window that rate and increase calculations use (1m to 1h). A longer window gives smoother lines. A shorter window shows short spikes.
Kafka Partition	Limit the Kafka panels to one or more partitions.

Some rows apply only to some pipelines. For example, the Solana Source row has data only for Solana sources. A panel that shows No data usually means that the pipeline does not use that feature. It does not mean that there is a problem.

Pipeline overview

General throughput, latency, and lag for each component.

Panel	What it shows	Use it to
Component Summary - Total Records	A table of each component with its node type, operator type, and total input and output records since the pipeline started.	Confirm that records reach every component. A transform whose output is much lower than its input is filtering records.
Cumulative Records Processed	Total input and output records for each component over time.	Find the time when a component stopped receiving or sending records (the line goes flat).
Overall Pipeline Throughput (records/sec)	Output records per second for each component.	Compare throughput between components and find the slowest one.
Avg Batch Size by Component	Average number of records in each batch that a component outputs.	Check how full batches are. Many small batches cause more overhead than fewer large batches.
Record Batch Processing Latency (p50, p95)	Time that each component takes to process one batch, at the 50th and 95th percentile.	Find a slow transform or sink. A high p95 with a normal p50 means that some batches are much slower than others.
Record Batch Average Processing Time	Average time that each component takes to process one batch.	See the general trend of processing time, with less noise than the percentile panel.
Block Lag (Max)	How far behind the chain tip the pipeline is, in seconds, over time.	Find out when the pipeline started to fall behind, and whether it is catching up.
Block Lag Gauge	The current block lag, in seconds.	See the current lag at a glance.

Block lag is available only for pipelines whose output includes a block number or timestamp column. Block lag is an end-to-end metric: a slow sink can cause it to grow. For more information, see Block lag on the Health overview page.

Checkpoints

A checkpoint saves the position of the pipeline after all sinks confirm that they wrote a set of records. These panels show how long that takes and whether it succeeds. For more information about checkpoints, see Delivery guarantees.

Panel	What it shows	Use it to
Checkpoint Epochs	Number of checkpoints that succeeded and failed in each time window.	Detect failed checkpoints. Any failed checkpoint needs investigation.
Checkpoint Epochs In Flight	Number of checkpoints in progress.	Check that checkpoints complete. A value that stays above zero for a long time means that a checkpoint is stuck, usually because a sink did not confirm its write.
Checkpoint Epoch Duration (p50, p95)	Time from the start of a checkpoint to its completion.	Measure how long records stay in the pipeline before the pipeline confirms delivery.
Checkpoint Sink Flush (p50, p95)	Time that each sink takes to write its data and confirm a checkpoint.	Find the sink that makes checkpoints slow.
Checkpoint Messages	Number of checkpoint markers sent, confirmations (acks) received, and finalizations sent.	Check that every marker gets a confirmation. If acks received stays lower than markers sent, a sink is not confirming.
Checkpoint Marker Arrival (p50, p95)	Time that a checkpoint marker takes to move through the pipeline and arrive at each component.	Find a component that delays other components. A slow component upstream makes markers arrive late at all components after it.

If Checkpoint Epoch Duration is high, compare Checkpoint Sink Flush and Checkpoint Marker Arrival:

High sink flush: the sink writes slowly. Check the destination database or service.
High marker arrival: records wait upstream of the sink. Check the latency of transforms, or the batch settings.

Kafka

Metrics for components that read from Kafka (consumer) or write to Kafka (producer). EVM dataset sources in streaming mode read from Kafka. The producer panels have data only if the pipeline has a Kafka sink.

Panel	What it shows	Use it to
Kafka Consumer Lag (Aggregated)	Number of messages that the source has not read yet, summed across all partitions.	Check whether the source keeps up with new data. Lag that increases steadily means that the pipeline falls behind.
Kafka Consumer Lag by Partition	Number of unread messages for each partition.	Find a single partition that falls behind while the others keep up.
Kafka Consumer Message Decode Latency (p50, p95)	Time to decode messages read from Kafka.	Find out if decoding is the bottleneck for a source.
Kafka Producer Message Encode Latency (p50, p95)	Time to encode messages before the sink sends them to Kafka.	Find out if encoding is the bottleneck for a Kafka sink.
Kafka Producer Batch Send Latency (p50, p95)	Time to send a batch of messages to the Kafka broker.	Find a slow broker or network problems between the pipeline and your Kafka cluster.
Kafka Consumer Row Kind Count Rate	Number of records read from Kafka, by row kind (insert, update, or delete).	See the mix of operations that the source receives.

High Kafka lag during a backfill is normal. For more information, see Pipeline status on the Health overview page.

Solana source

These panels have data only for pipelines with a Solana source.

Panel	What it shows	Use it to
Block Rate	Number of blocks per second that the source processes.	Check the speed of the source.
Buffer Size	Number of blocks in the source’s internal buffer.	Detect backpressure. A buffer that grows means that the rest of the pipeline cannot process blocks as fast as the source fetches them.
Block Slot	The next Solana slot that the source will process.	Track progress. A flat line means that the source has stopped.
Fetch Duration	Time to fetch data from the Solana data source, at the 95th percentile.	Find out if the slow part is the fetch itself.

Handler/Webhook HTTP metrics

These panels have data only for pipelines with an HTTP handler transform or a webhook sink.

Panel	What it shows	Use it to
Requests per minute by status	Number of HTTP requests per minute, by status class: `2xx`, `4xx`, `5xx`, or, when no response arrives, `timeout` or `network_error`.	Find errors that your endpoint returns, and requests that do not reach it.
Requests per minute by outcome	Number of HTTP requests per minute, by outcome: `success`, `retriable_error` (the pipeline tries the request again), or `non_retriable_error`.	Find out if errors are temporary or permanent. A steady rate of `non_retriable_error` usually means a configuration problem, such as a wrong URL or bad credentials.
Request Latency	Response time of your endpoint at the 95th and 99th percentile.	Find out if a slow endpoint limits the pipeline.

Kubernetes

Resource usage of the containers that run the pipeline.

Panel	What it shows	Use it to
Pod Status by Phase	The phase of each pipeline pod over time (for example `Running` or `Pending`).	Detect rescheduling. A new pod name, or a phase other than `Running`, means that the pipeline moved to a new pod.
CPU Usage vs Provisioned	CPU use, compared to the CPU request and limit.	Find out if the pipeline needs more CPU. Usage at the limit for a long time means that you can increase `resource_size`.
Memory Usage vs Provisioned	Memory use, compared to the memory request and limit.	Find out if the pipeline is near its memory limit. A pod that reaches its memory limit restarts.
Network Bytes (RX/TX)	Bytes received and sent per second.	Compare network traffic with throughput, and find unexpected traffic.
Network connections	The highest number of connections opened, closed, failed, and refused in each time window.	Find connection problems with a source or sink. Failed or refused connections usually mean that a destination is not reachable.
Network Dropped Packets (RX/TX)	Packets received and sent that were dropped.	Find network problems. This value is usually zero.

Next steps

Use the Health overview to monitor all pipelines in your project.
Set up custom alerts on these metrics.
Send the same metrics to your own monitoring system with the Prometheus integration.

Turbo

Guides

Reference

Mirror (legacy)

Advanced metrics dashboard

Overview

Filters

Pipeline overview

Checkpoints

Kafka

Solana source

Handler/Webhook HTTP metrics

Kubernetes

Next steps

​Overview

​Filters

​Pipeline overview

​Checkpoints

​Kafka

​Solana source

​Handler/Webhook HTTP metrics

​Kubernetes

​Next steps

Overview

Filters

Pipeline overview

Checkpoints

Kafka

Solana source

Handler/Webhook HTTP metrics

Kubernetes

Next steps