Choosing a path
You have two options to create a Goldsky Mirror pipeline:
- Interactively: With a guided CLI experience
- Non-interactively: By providing a pipeline configuration
In both cases, you use the
goldsky pipeline create CLI command.
Interactive: Guided CLI experience
This is the simplest way to create a new pipeline. Run
goldsky pipeline create <your-pipeline-name> in your terminal and follow the prompts.
In short, the CLI guides you through the following process:
- Select one or more source(s)
- Depending on the selected source(s), define transforms
- Configure one or more sink(s)
Non-interactive: Pipeline definition
This is an advanced way to create a new pipeline. Instead of using the guided CLI experience (see above), you create the pipeline definition on your own. Run
goldsky pipeline create <your-pipeline-name> --definition-path ./pipeline.yaml in your terminal to create a pipeline.
Writing a definition file
A pipeline definition is a YAML structure with the following top-level properties:
sources:  transforms:  sinks: 
sinks are required with a minimum of one entry each.
transforms is optional; enter an empty array if no transforms are needed.
Click through here for the full reference documentation on Mirror pipeline configuration files, or to the source and sink-specific documentation. Note that to create a pipeline from definition, you need to have a database secret already configured on your Goldsky project.
Starting a pipeline from definition
After setting up a secret, you start a pipeline with:
goldsky pipeline create edge-logs --definition-path <path_to_file>
Monitor a pipeline
When you create a new pipeline, the CLI automatically starts to monitor the status and outputs it in a table format.
If you want to monitor an existing pipeline at a later time, use the
goldsky pipeline monitor <your-pipeline-name> CLI command. It refreshes every ten seconds and gives you insights into how your pipeline performs.