Skip to main content

​
Deploying a pipeline

There are two main ways by which you can deploy a pipeline: in the web app or by using the CLI.
If you prefer to deploy pipelines using a web interface instead check the Pipeline Builder

​
apply command + pipeline configuration

The goldsky pipeline apply command expects a pipeline configuration file. For example:
base-logs.yaml
Please save the configuration in a file and run goldsky pipeline apply <path_to_config_file> --status ACTIVE to deploy the pipeline.

​
Pausing a pipeline

There are several ways by which you can pause a pipeline:

​
1. pause command

goldsky pipeline pause <name> will attempt to take a snapshot before pausing the pipeline. The snapshot is successfully taken only if the pipeline is in a healthy state. After snapshot completes, the pipeline desired status to PAUSED runtime status to TERMINATED. Example:

​
2. stop command

You can stop a pipeline using the command goldsky pipeline stop <name>. Unlike the pause command, stopping a pipeline doesn’t try to take a snapshot. Mirror will directly set pipeline to INACTIVE desired status and TERMINATED runtime status. Example:

​
3. apply command + INACTIVE or PAUSED status

We can replicate the behaviour of the pause and stop commands using pipeline apply and setting the --status flag to INACTIVE or PAUSED. Following up with our previous example, we could stop our deployed pipeline with goldsky pipeline apply <name_or_path_to_config> --status INACTIVE

​
Restarting a pipeline

There are two ways to restart an already deployed pipeline:

​
1. restart command

As in: goldsky pipeline restart <name> --from-snapshot last|none Example:
This command will open up a monitor for your pipeline after deploying.

​
2. apply command + ACTIVE status

Just as you can stop a pipeline changing its status to INACTIVE you can also restart it by setting it to ACTIVE Following up with our previous example, we could restart our stopped pipeline with goldsky pipeline apply base-logs-pipeline --status ACTIVE
Unlike the start command, this method won’t open up the monitor automatically.

​
Applying updates to pipeline configuration

For example:
base-logs.yaml
In this example we are changing the pipeline description and resource_size of the pipeline using its latest succesful snapshot available and informing Mirror to not take a snapshot before applying the update. This is a common configuration to apply in a situation where you found issues with your pipeline and would like to restart from the last healthy checkpoint. For a more complete reference on the configuration attributes you can apply check this reference.

​
Deleting a pipeline

Although pipelines with desired status INACTIVE don’t consume any resources (and thus, do not imply a billing cost on your side) it’s always nice to keep your project clean and remove pipelines which you aren’t going to use any longer. You can delete pipelines with the command goldsky pipeline delete:

​
In-flight requests

Sometimes you might experience that you are not able to perform a specific action on your pipeline because an in-flight request is currently being processed. What this means is that there was a previous operation performed in your pipeline which hasn’t finished yet and needs to be either processed or discarded before you can apply your specific operation. A common scenario for this is your pipeline is busy taking a snapshot. Consider the following example where we recently paused a pipeline (thus triggering a snapshot) and we immediately try to delete it:
Let’s look at what process is still to be processed:
We can see that the snapshot is still taking place. Since we want to delete the pipeline we can go ahead and stop this snapshot creation:
We can now succesfully remove the pipeline:
As you saw in this example, Mirror provides you with commands to see the current in-flight requests in your pipeline and decide whether you want to discard them or wait for them to be processed.