Choosing a path

You have two options to create a Goldsky Mirror pipeline:

  1. Interactively: With a guided CLI experience
  2. Non-interactively: By providing a pipeline configuration

In both cases, you use the goldsky pipeline create CLI command.

Interactive: Guided CLI experience

This is the simplest way to create a new pipeline. Run goldsky pipeline create <your-pipeline-name> in your terminal and follow the prompts.

In short, the CLI guides you through the following process:

  1. Select one or more source(s)
  2. Depending on the selected source(s), define transforms
  3. Configure one or more sink(s)

Non-interactive: Pipeline definition

This is an advanced way to create a new pipeline. Instead of using the guided CLI experience (see above), you create the pipeline definition on your own. Run goldsky pipeline create <your-pipeline-name> --definition-path ./pipeline.yaml in your terminal to create a pipeline.

Writing a definition file

A pipeline definition is a YAML structure with the following top-level properties:

sources: []
transforms: []
sinks: []

Both sources and sinks are required with a minimum of one entry each. transforms is optional; enter an empty array if no transforms are needed.

Click through here for the full reference documentation on Mirror pipeline configuration files, or to the source and sink-specific documentation. Note that to create a pipeline from definition, you need to have a database secret already configured on your Goldsky project.

Starting a pipeline from definition

After setting up a secret, you start a pipeline with:

goldsky pipeline create edge-logs --definition-path <path_to_file>

Monitor a pipeline

When you create a new pipeline, the CLI automatically starts to monitor the status and outputs it in a table format.

If you want to monitor an existing pipeline at a later time, use the goldsky pipeline monitor <your-pipeline-name> CLI command. It refreshes every ten seconds and gives you insights into how your pipeline performs.