Pipelines

The table below describes the list of actions that can be performed on "pipelines" using the RCTL CLI Utility.

Resource	Create	Get	Update	Delete
pipeline	YES	YES	YES	YES

---

Create Pipeline

A declarative approach to lifecycle management of pipelines is strongly recommended backed by the pipeline manifests (YAML files) that are version controlled in your Git repository.

---

Declarative

You can create/update a pipeline in the project in your Organization based on a version controlled pipeline spec that you can store in a Git repository. This enables users to develop automation for reproducible infrastructure.

./rctl create pipeline -f <pipeline-file.yaml> --v3

(or)

./rctl apply -f pipeline-spec.yml

Pipeline Config Spec

Here's an illustrative example of the pipeline spec YAML file for various actions

Approval

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  modifiedAt: "2023-11-06T13:58:48.226056Z"
  name: approval-type
  project: project-a
spec:
  active: true
  stages:
  - config:
      approvers:
      - userName: demo@rafay.co
      timeout: 0s
      type: Email
    name: demo1
    type: Approval
  triggers:
  - config:
      repo:
        provider: Github
        repository: repo-01
        revision: master
    name: demo2
    type: Webhook

Deploy Workload

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  modifiedAt: "2023-11-06T14:05:36.788293Z"
  name: deploy-workload
  project: project-a
spec:
  stages:
  - config:
      workload: workload-02
    name: demo1
    type: DeployWorkload
  triggers:
  - config:
      repo:
        provider: Github
        repository: repo-01
        revision: main
    name: demo2
    type: Webhook

Deploy Workload Template

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  name: test-pipeline
  project: defaultproject
spec:
  active: true
  sharing:
    enabled: false
  stages:
  - config:
      namespace: ns1
      placement:
        labels:
        - key: rafay.dev/clusterName
          value: pipeline-k3d-1
      workloadTemplate: nginx-helm
      projectName: default
      workloadName: demo-workload
    name: stage1
    type: DeployWorkloadTemplate

Infrastructure Provisioner

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  modifiedAt: "2022-07-14T06:55:18.304998Z"
  name: demo_pipeline
  project: gitops-workloads
spec:
  stages:
  - config:
      action:
        action: Plan
        refresh: true
        version: 0.14.9
      agents:
      - name: demo-agent
      provisioner: manual
      type: Terraform
    name: t1
    next:
    - name: apply
    type: InfraProvisioner
  - config:
      action:
        action: Apply
        refresh: true
      agents:
      - name: demo-agent
      provisioner: manual
      type: Terraform
    name: apply
    type: InfraProvisioner

System Sync

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  name: test-pipe
  project: prod-test
spec:
  active: true
  stages:
  - config:
      destinationRepo: {}
      gitToSystemSync: true
      includedResources:
      - name: '*'
      sourceAsDestination: true
      sourceRepo:
        path:
          name: prod-sanity-check-shared-repo
        repository: test-project
        revision: main
      systemToGitSync: true
    name: system-sync-stage
    type: SystemSync
  triggers:
  - config:
      repo:
        paths:
        - name: prod-sanity-check
        provider: Github
        repository: test-repo
        revision: main
    name: prod-sanity
    type: Webhook

---

Pipeline Execution with Parallel Stage Support

To improve pipeline efficiency and reduce deployment time, support is being introduced for parallel execution of independent 'Deploy Workload' stages. This enhancement allows stages within the same pipeline to run simultaneously when no dependencies exist between them.

To enable parallel execution and define stage dependencies, the following configuration options are introduced in the pipeline specification:

The example below demonstrates how a pipeline can be structured to execute independent stages in parallel. The pipeline is designed to allow multiple approval stages to run simultaneously, with a final stage that aggregates the results of these parallel processes.

• Sequential Preparation: The stages initialApproval, stageSplitter, and prepareParallelStages execute sequentially to set up the parallel workflow.

• Parallel Stage Execution: The stages parallelTaskA, parallelTaskB, parallelTaskC, and parallelTaskD are independent and execute in parallel. No explicit dependencies are declared between them, allowing for concurrent processing.

• Convergence and Synchronization: Each parallel task transitions to a common stage, finalAggregation, which runs only after all parallel stages complete successfully.

❗ Important Note
If any stage in the pipeline fails—whether part of the sequential flow or a parallel branch—the entire pipeline is marked as failed. This ensures a consistent failure model and prevents partial execution from proceeding to downstream stages.

apiVersion: gitops.k8smgmt.io/v3
kind: Pipeline
metadata:
  name: parallel-processing-pipeline
  project: defaultproject
spec:
  stages:
  - name: initialApproval
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: stageSplitter
  - name: stageSplitter
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: prepareParallelStages
  - name: prepareParallelStages
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: parallelTaskA
    - name: parallelTaskB
    - name: parallelTaskC
    - name: parallelTaskD
  - name: parallelTaskA
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: finalAggregation
  - name: parallelTaskB
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: finalAggregation
  - name: parallelTaskC
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: finalAggregation
  - name: parallelTaskD
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email
    next:
    - name: finalAggregation
  - name: finalAggregation
    type: Approval
    config:
      approvers:
      - ssoUser: true
        userName: demouser@test.com
      type: Email

---

List Pipelines

Use this command to retrieve the list of pipelines in the configured Project. An illustrative example is shown below where RCTL retrieves the list of pipelines in the project "qa".

./rctl get pipeline -p qa

+---------------+-----------+-------------------------------------------+--------------------------------+------------------------------+
| PIPELINE NAME | STATUS    | STAGES                                    | EDGES(SOURCE-TARGET)           | CREATE AT                    |
+---------------+-----------+-------------------------------------------+--------------------------------+------------------------------+
| nginx         | Activated | deploy_nginx_workload:DeployWorkloadStage |                                | Sun Dec 20 23:09:00 UTC 2020 |
+---------------+-----------+-------------------------------------------+--------------------------------+------------------------------+
| multitenant   | Activated | approval:ApprovalStage,                   | approval-tenant-a,             | Mon Dec 21 19:43:40 UTC 2020 |
|               |           | tenant-a:DeployWorkloadStage,             | tenant-a-tenant-b,             |                              |
|               |           | tenant-b:DeployWorkloadStage,             | tenant-b-tenant-c              |                              |
|               |           | tenant-c:DeployWorkloadStage              |                                |                              |
+---------------+-----------+-------------------------------------------+--------------------------------+------------------------------+

---

Get The Specific Pipeline Info

Use this command to retrieve a specific pipeline's details in the configured project.

./rctl get pipeline <pipeline-name> -p <project-name>

Below is the illustrative example for the pipeline called "nginx"

./rctl get pipeline nginx

+---------------+-----------+-------------------------------------------+----------------------+------------------------------+
| PIPELINE NAME | STATUS    | STAGES                                    | EDGES(SOURCE-TARGET) | CREATE AT                    |
+---------------+-----------+-------------------------------------------+----------------------+------------------------------+
| nginx         | Activated | deploy_nginx_workload:DeployWorkloadStage |                      | Sun Dec 20 23:09:00 UTC 2020 |
+---------------+-----------+-------------------------------------------+----------------------+------------------------------+

Or you can use below command to get more information of the addon in json or yaml format

./rctl get pipeline <pipeline-name> -o json

./rctl get pipeline <pipeline-name> -o yaml

---

Delete Pipeline

You can delete a pipeline in the configured project.

./rctl delete pipeline -f <filename> --v3

---

Activate/Deactivate Pipeline

To activate/deactivate the pipeline using the config file, update the below parameters with true/false and run the apply command

spec:
  active: true/false

Also, you can use the below command to activate/deactivate the pipeline without updating the spec config

./rctl activate pipeline <name> --v3

and

./rctl deactivate pipeline <name> --v3

Applying these two commands will automatically update the spec file parameter active: true/false

---

Start Pipeline

To start the pipeline, use the below command

./rctl start pipeline <pipeline_name>

---

Stop Pipeline

To stop the pipeline, use the below command

./rctl stop pipeline <pipeline_name>

Important

Refer here for the deprecated RCTL Commands

---