Data quality

​

Indexing

Goldsky datasets are populated through various indexers, which write the data into a data stream. The data stream is accessible directly by users through Turbo and legacy Mirror pipelines. Internally, we copy the data to a data lake which is then used to power various features and also used for data QA. Data quality is managed during ingestion, and also through periodic checks. Emitted data quality is managed through various database guarantees, depending on the destination of the data.

​

Ingestion-level consistency

​

Chain continuity

When first ingesting a block, we check for a continuous block hash chain. If the chain is not valid (i.e. the parent hash does not match the hash we have of the preceding block number), we issue deletes and updates into our dataset and walk backwards until we reach a consistent chain again. All deletes and updates are propagated through to downstream sinks. This means if you have a pipeline writing chain data into a database, and that chain goes through a reorg or a rollback, all the changes will automatically propagate to your database as well.

​

Write guarantees

During ingestion, we ensure we have the full set of data for a block before emitting it into the various datasets. When emitting, we acquire full consistency acknowledgement from our various data sinks before marking the block as ingested.

​

Schema strictness

Our datasets follow strict typed schemas, causing writes that don’t fit into said schemas to fail completely.

​

Dataset validation checks

In rare cases, RPC nodes can give us invalid data that may be missed during ingestion checks. For every dataset, we run checks on a daily basis and repair the data if any issues are seen. These checks validate:

1. Missing blocks (EVM) - we record the minimum and maximum block numbers for each date, and look for gaps in the data
2. Missing transactions (EVM) - We count unique transaction hashes per block and compare it with the transaction_count for the block.
3. Missing logs (EVM) - We compare the maximum log index per block with the number of logs per block.

These checks work like unit tests: write one for a chain, and it runs against every chain from then on, catching regressions before you see them in your data.

​

Destination-level consistency

To prevent missing data when writing, both Turbo and Mirror pipelines are built with an at-least-once guarantee. Turbo pipelines coordinate source commits and sink flushes through a checkpointing protocol: a sink writes records to its destination first, and only after every sink confirms its write does the source commit its read position. See Delivery guarantees for how the protocol works, when duplicates can occur, and how to design idempotent sinks.

​

Mirror (legacy) consistency

Mirror pipelines achieve the same at-least-once guarantee with periodic snapshots instead of checkpoints:

• Snapshots: automatic fault tolerance every minute with snapshot recovery every 4 hours. When a pipeline is updated or forced to terminate, a snapshot is persisted and used for the next incarnation of the pipeline, preserving continuity of the data being sent.
• Database acknowledgement: every row requires full acknowledgement from the database before the pipeline moves to the next batch. Unacknowledged data is not marked sent in the snapshot, so on restart the pipeline is pessimistic and risks resending data over missing it.
• Sink downtime handling: a failed write retries just that batch for that sink, then restarts the writers, and after prolonged failure the pipeline fails and resumes from the last saved snapshot when restarted.