MAINNET SUPPORTED TESTNET SUPPORTED
Turbo pipelines provide high-performance streaming data pipelines with sub-second latency. Deploy a pipeline to start streaming Stellar data to your preferred destination. ### Quick config ```yaml stellar-transactions.yaml theme={null} name: my-stellar-transactions resource_size: s sources: stellar_transactions: type: dataset dataset_name: stellar_mainnet.transactions version: 1.2.0 start_at: latest sinks: my_sink: type: postgres from: stellar_transactions schema: public table: stellar_transactions secret_name: MY_POSTGRES_SECRET primary_key: transaction_hash ``` Deploy with: ```bash theme={null} goldsky turbo apply stellar-transactions.yaml ``` ### Available chain slugs Mainnet: `stellar_mainnet` | Testnet: `stellar_testnet` Goldsky provides real-time (under 5 seconds) streaming of Stellar datasets, including all historical data, for both mainnet and testnet. Datasets include ledgers, transactions, operations, events, transfers, ledger entries, and balances.
```mermaid theme={null}
stateDiagram-v2
state ACTIVE {
[*] --> STARTING
STARTING --> RUNNING
}
```
Let's look at a simple example below where we configure a pipeline that consumes Logs from Base chain and streams them into a Postgres database:
```yaml base-logs.yaml theme={null}
name: base-logs-pipeline
resource_size: s
apiVersion: 3
sources:
base.logs:
dataset_name: base.logs
version: 1.0.0
type: dataset
description: Enriched logs for events emitted from contracts. Contains the contract address, data, topics, decoded event and metadata for blocks and transactions.
display_name: Logs
transforms: {}
sinks:
postgres_base_logs:
type: postgres
table: base_logs
schema: public
secret_name: GOLDSKY_SECRET
description: "Postgres sink for: base.logs"
from: base.logs
```
Let's attempt to run it using the command `goldsky pipeline apply base-logs.yaml --status ACTIVE` or `goldsky pipeline start base-logs.yaml`
```
❯ goldsky pipeline apply base-logs.yaml --status ACTIVE
│
◇ Successfully validated config file
│
◇ Successfully applied config to pipeline: base-logs-pipeline
To monitor the status of your pipeline:
Using the CLI: `goldsky pipeline monitor base-logs-pipeline`
Using the dashboard: https://app.goldsky.com/dashboard/pipelines/stream/base-logs-pipeline/1
```
At this point we have set the desired status to `ACTIVE`. We can confirm this using `goldsky pipeline list`:
```
❯ goldsky pipeline list
✔ Listing pipelines
────────────────────────────────────────
│ Name │ Version │ Status │ Resource │
│ │ │ │ Size │
│───────────────────────────────────────
│ base-logs-pipeline │ 1 │ ACTIVE │ s │
────────────────────────────────────────
```
We can then check the runtime status of this pipeline using the `goldsky pipeline monitor base-logs-pipeline` command:
```mermaid theme={null}
---
title: Healthy pipeline becomes unhealthy
---
stateDiagram-v2
state status:ACTIVE {
[*] --> STARTING
STARTING --> RUNNING
RUNNING --> FAILING
FAILING --> TERMINATED
}
```
```mermaid theme={null}
---
title: Pipeline cannot start
---
stateDiagram-v2
state status:ACTIVE {
[*] --> STARTING
STARTING --> FAILING
FAILING --> TERMINATED
}
```
A Pipeline can be in an `ACTIVE` desired status but a `TERMINATED` runtime status in scenarios that lead to terminal failure.
Let's see an example where we'll use the same configuration as above but set a `secret_name` that does not exist.
```yaml bad-base-logs.yaml theme={null}
name: bad-base-logs-pipeline
resource_size: s
apiVersion: 3
sources:
base.logs:
dataset_name: base.logs
version: 1.0.0
type: dataset
description: Enriched logs for events emitted from contracts. Contains the contract address, data, topics, decoded event and metadata for blocks and transactions.
display_name: Logs
transforms: {}
sinks:
postgres_base_logs:
type: postgres
table: base_logs
schema: public
secret_name: YOUR_DATABASE_SECRET
description: "Postgres sink for: base.logs"
from: base.logs
```
Let's start it using the command `goldsky pipeline apply bad-base-logs.yaml`.
```
❯ goldsky pipeline apply bad-base-logs.yaml
│
◇ Successfully validated config file
│
◇ Successfully applied config to pipeline: base-logs-pipeline
To monitor the status of your pipeline:
Using the CLI: `goldsky pipeline monitor bad-base-logs-pipeline`
Using the dashboard: https://app.goldsky.com/dashboard/pipelines/stream/bad-base-logs-pipeline/1
```
The pipeline configuration is valid, however, the pipeline runtime will encounter error since the secret that contains credentials to communicate with the sink does not exist.
Running `goldsky pipeline monitor bad-base-logs-pipeline` we see: