Skip to content

Cluster Overrides

By default, K8s objects require certain values be set inside their specs that match their cluster's configuration. If we were to do this for a single addon across a fleet of clusters we could have many duplicate addons. To mitigate this the platform supports cluster overrides. These allow the customer to define values that will be applied to K8s objects defined in addons as they are deployed via the controller. This allows the customer to use a single addon org-wide and override the values when addons are deployed.

Platform resources such as addons and workloads utilize labels which are added by the platform or by the end user. These labels are used as selectors for both the resource they will be applied as well as the clusters where the cluster override will be applied.

Important

You can manage the lifecycle of cluster overrides using the Web Console or RCTL CLI or REST APIs. It is strongly recommended to automate this by integrating RCTL with your existing CI system based automation pipeline.


In the example below we are using the AWS Load Balancer Controller addon we have created through the platform. The addon called "aws-lb-controller" is configured with the helm chart provided by AWS. The helm chart requires that a "clusterName" be set before being deployed. By default the value is left blank inside the chart. We will utilize a cluster override to set the value when the helm chart is deployed.


Step 1: Create Cluster Override

As an Admin in the Web Console,

  • Navigate to the Project
  • Click on Cluster Overrides under Infrastructure
  • Click on New Override
  • Provide a name and select the appropriate Cluster Override Type

New Override


Step 2: Edit the Cluster Override

  • Click on the newly created Cluster Override
  • For the Resource Selector select the addon for which the cluster override will be applied
  • For the Type select Specific Clusters and select the cluster for which the cluster override will be applied
  • Optionally Customer Labels and Custom Values are supported as a Placement type and can be used as a selector
  • Add the clusterName and value to the Override Values
  • Save Changes
clusterName: my-cluster-name

New Override Config


Step 3: Deploy the Addon

  • Deploy the blueprint to the cluster containing the addon utilizing the newly created cluster override

Step 4: Verify the cluster override has been applied to the deployment

kubectl describe pod -n kube-system aws-lb-controller-aws-load-balancer-controller-f5f6d6b47-9kjkl

Name:         aws-lb-controller-aws-load-balancer-controller-f5f6d6b47-9kjkl
Namespace:    kube-system
Priority:     0
Node:         ip-172-31-114-123.us-west-1.compute.internal/172.31.114.123
Start Time:   Wed, 07 Jul 2021 16:33:19 +0000
Labels:       app.kubernetes.io/instance=aws-lb-controller
              app.kubernetes.io/name=aws-load-balancer-controller
              pod-template-hash=f5f6d6b47
              rep-addon=aws-lb-controller
              rep-cluster=pk0d152
              rep-drift-reconcillation=enabled
              rep-organization=d2w714k
              rep-partner=rx28oml
              rep-placement=k69rynk
              rep-project=lk5rdw2
              rep-workloadid=kv6p0vm
Annotations:  kubernetes.io/psp: rafay-kube-system-psp
              prometheus.io/port: 8080
              prometheus.io/scrape: true
Status:       Running
IP:           172.31.103.206
IPs:
  IP:           172.31.103.206
Controlled By:  ReplicaSet/aws-lb-controller-aws-load-balancer-controller-f5f6d6b47
Containers:
  aws-load-balancer-controller:
    Container ID:  docker://e115a56b7444ea55bda8f2503b9b046d6fd84dbffd3cbf77090f35f35c2657ef
    Image:         602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.1.3
    Image ID:      docker-pullable://602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller@sha256:c7981cc4bb73a9ef5d788a378db302c07905ede035d4a529bfc3afe18b7120ef
    Ports:         9443/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /controller
    Args:
      --cluster-name=my-cluster-name
      --ingress-class=alb
    State:          Running
      Started:      Wed, 07 Jul 2021 16:33:25 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:61779/healthz delay=30s timeout=10s period=10s #success=1 #failure=2
    Environment:    <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from aws-lb-controller-aws-load-balancer-controller-token-dllmd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  aws-load-balancer-tls
    Optional:    false
  aws-lb-controller-aws-load-balancer-controller-token-dllmd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  aws-lb-controller-aws-load-balancer-controller-token-dllmd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m27s  default-scheduler  Successfully assigned kube-system/aws-lb-controller-aws-load-balancer-controller-f5f6d6b47-9kjkl to ip-172-31-114-123.us-west-1.compute.internal
  Normal  Pulling    3m26s  kubelet            Pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.1.3"
  Normal  Pulled     3m22s  kubelet            Successfully pulled image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.1.3" in 4.252275657s
  Normal  Created    3m21s  kubelet            Created container aws-load-balancer-controller
  Normal  Started    3m21s  kubelet            Started container aws-load-balancer-controller

Example with Custom Labels Placement Type

New Override Config

Override Values

clusterName: my-cluster-name

Example with Custom Value Placement Type

New Override Config

Override Values

replicaCount: 1
image:
  repository: nginx
  pullPolicy: Always
  tag: "1.19.8"
service:
  type: ClusterIP
  port: 8080

Cluster Overrides using the CLI

Cluster Overrides for Workloads