How to build real-time Bitcoin data pipelines for blocks and transactions

Overview

Goldsky provides curated Bitcoin datasets for blocks and transactions, making it easy to build pipelines for Bitcoin blockchain analysis.

Quick Start

Get started with Bitcoin data - choose the dataset that fits your use case: For block data:

sources:
  bitcoin_blocks:
    type: dataset
    dataset_name: bitcoin.raw.blocks
    version: 1.0.0
    start_at: latest

For transaction data:

sources:
  bitcoin_transactions:
    type: dataset
    dataset_name: bitcoin.raw.transactions
    version: 1.0.0
    start_at: latest

Available datasets:

bitcoin.raw.blocks - Block data including hash, height (number), size, weight, merkle root, timestamp, and transaction count
bitcoin.raw.transactions - Transaction data including inputs, outputs (as JSON strings), input/output counts and values, fee, and block metadata

Schema Reference

bitcoin.raw.blocks - primary key id

Field	Type	Description
`id`	STRING	Unique row id (primary key)
`hash`	STRING	Block hash
`size`	LONG	Block size in bytes
`stripped_size`	LONG	Block size without witness data
`weight`	LONG	Block weight
`number`	LONG	Block height
`version`	LONG	Block version
`merkle_root`	STRING	Merkle root hash
`timestamp`	LONG	Block time (Unix seconds)
`nonce`	STRING	Block nonce (hex)
`bits`	STRING	Difficulty target in compact form
`coinbase_param`	STRING	Coinbase parameter
`transaction_count`	LONG	Number of transactions in the block

bitcoin.raw.transactions - primary key id

Field	Type	Description
`id`	STRING	Unique row id (primary key)
`hash`	STRING	Transaction hash (`txid`)
`size`	LONG	Transaction size in bytes
`virtual_size`	LONG	Virtual size (vsize)
`version`	LONG	Transaction version
`lock_time`	LONG	Lock time
`block_number`	LONG	Block height containing this tx
`block_hash`	STRING	Hash of the containing block
`block_timestamp`	LONG	Block time (Unix seconds)
`is_coinbase`	BOOLEAN	Whether this is a coinbase tx
`index`	LONG	Index within the block
`inputs`	STRING	JSON-encoded array of inputs
`outputs`	STRING	JSON-encoded array of outputs
`input_count`	LONG	Number of inputs
`output_count`	LONG	Number of outputs
`input_value`	LONG	Total input value in satoshis
`output_value`	LONG	Total output value in satoshis
`fee`	LONG	Transaction fee in satoshis

inputs and outputs are JSON-encoded strings. Use json_extract or your sink’s JSON functions to parse them downstream.

Guide: Track Large Bitcoin Transactions

Monitor transactions with significant BTC movement:

name: bitcoin-whale-tracker
resource_size: s

sources:
  transactions:
    type: dataset
    dataset_name: bitcoin.raw.transactions
    version: 1.0.0
    start_at: latest

transforms:
  # Filter to large transactions (>1 BTC = 100,000,000 satoshis)
  large_transactions:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        hash,
        block_hash,
        block_number,
        block_timestamp,
        input_value,
        output_value,
        fee,
        _gs_op
      FROM transactions
      WHERE output_value > 100000000

sinks:
  postgres_whales:
    type: postgres
    from: large_transactions
    schema: public
    table: bitcoin_whale_transactions
    secret_name: MY_POSTGRES
    primary_key: id

Guide: Monitor Block Production

Track Bitcoin block metadata for network analysis:

name: bitcoin-block-monitor
resource_size: s

sources:
  blocks:
    type: dataset
    dataset_name: bitcoin.raw.blocks
    version: 1.0.0
    start_at: latest

transforms:
  block_stats:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        hash,
        number,
        timestamp,
        transaction_count,
        size,
        stripped_size,
        weight,
        merkle_root,
        _gs_op
      FROM blocks

sinks:
  postgres_blocks:
    type: postgres
    from: block_stats
    schema: public
    table: bitcoin_blocks
    secret_name: MY_POSTGRES
    primary_key: id

Guide: Analyze Transaction Fees

Track Bitcoin transaction fees over time:

name: bitcoin-fee-tracker
resource_size: s

sources:
  transactions:
    type: dataset
    dataset_name: bitcoin.raw.transactions
    version: 1.0.0
    start_at: latest

transforms:
  fee_analysis:
    type: sql
    primary_key: id
    sql: |
      SELECT
        id,
        hash,
        block_number,
        block_timestamp,
        fee,
        size,
        virtual_size,
        CAST(fee AS DOUBLE) / CAST(size AS DOUBLE) as fee_per_byte,
        _gs_op
      FROM transactions
      WHERE fee > 0

sinks:
  postgres_fees:
    type: postgres
    from: fee_analysis
    schema: public
    table: bitcoin_fee_analysis
    secret_name: MY_POSTGRES
    primary_key: id

Performance Tips

Choose the right dataset

Use the most specific dataset for your use case:

Block-level analysis? Use bitcoin.raw.blocks
Transaction analysis? Use bitcoin.raw.transactions

Filter early

Apply filters in SQL as early as possible to reduce data volume:

transforms:
  filtered:
    type: sql
    sql: SELECT * FROM transactions WHERE output_value > 100000000

Use appropriate resource sizes

Bitcoin has moderate throughput compared to high-volume chains. Start with small or medium:

resource_size: s  # or m for historical backfills

Turbo

API

Reference

Bitcoin Sources

Overview

Quick Start

Schema Reference

Guide: Track Large Bitcoin Transactions

Guide: Monitor Block Production

Guide: Analyze Transaction Fees

Performance Tips

​Overview

​Quick Start

​Schema Reference

​Guide: Track Large Bitcoin Transactions

​Guide: Monitor Block Production

​Guide: Analyze Transaction Fees

​Performance Tips

Overview

Quick Start

Schema Reference

Guide: Track Large Bitcoin Transactions

Guide: Monitor Block Production

Guide: Analyze Transaction Fees

Performance Tips