Skip to content

Overview

Azure Kubernetes Service (AKS) is a fully managed Kubernetes service provided by Microsoft Azure. We have developed an integration with AKS to ensure that users can provision AKS Clusters in any region using the RCTL CLI.

Declarative

You can also create the cluster based on a version controlled cluster spec that you can store in a Git repository. This enables users to develop automation for reproducible infrastructure.

Create Cluster

./rctl create cluster aks -f cluster-spec.yaml

Use the below v1 Spec to create a AKS Cluster

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: aks-ui-01
  project: defaultproject
spec:
  blueprint: default-aks
  cloudprovider: provider-aks
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: aks-ui-01
    spec:
      managedCluster:
        apiVersion: "2021-05-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
          dnsPrefix: aks-ui-01-dns
          kubernetesVersion: 1.23.8
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2021-05-01"
        location: centralindia
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: System
          orchestratorVersion: 1.23.8
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2021-05-01"
        location: centralindia
        name: secondary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: User
          nodeLabels:
            app: infra
            dedicated: "true"
          nodeTaints:
          - app=infra:NoSchedule
          - dedicated=true:NoSchedule
          orchestratorVersion: 1.23.8
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo_resourcegroup
  proxyconfig: {}
  systemComponentsPlacement:
    nodeSelector:
      app: infra
      dedicated: "true"
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: "true"
    - effect: NoSchedule
      key: app
      operator: Equal
      value: infra
  type: aks

Successfully provisioned AKS cluster

Below is an example of the AKS addon yaml file

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: demo-aks-addon
  project: defaultproject
spec:
  blueprint: default-aks
  cloudprovider: azure-credentials
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: demo-aks-addon
    spec:
      managedCluster:
        apiVersion: "2022-07-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          addonProfiles:
            azureKeyVaultSecretsProvider:
              config:
                enableSecretRotation: "true"
                rotationPollInterval: 1m
              enabled: true
            azurePolicy:
              enabled: true
            httpApplicationRouting:
              enabled: true
            ingressApplicationGateway:
              config:
                applicationGatewayId: "Application Gateway ID"
              enabled: true
            omsAgent:
              config:
                logAnalyticsWorkspaceResourceID: "Resource ID"
              enabled: true
            openServiceMesh:
              enabled: true
          apiServerAccessProfile:
            enablePrivateCluster: false
          dnsPrefix: demo-aks-addon-dns
          kubernetesVersion: 1.22.11
          networkProfile:
            dnsServiceIP: 10.0.0.10
            dockerBridgeCidr: 172.17.0.1/16
            loadBalancerSku: standard
            networkPlugin: azure
            networkPolicy: azure
            serviceCidr: 10.0.0.0/16
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2022-07-01"
        location: centralindia
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: System
          nodeLabels:
            check1: value1
          nodeTaints:
          - check1=value1:PreferNoSchedule
          orchestratorVersion: 1.22.11
          osType: Linux
          tags:
            check1: value1
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2022-07-01"
        location: centralindia
        name: np1
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: User
          nodeLabels:
            check2: value2
          nodeTaints:
          - check2=value2:NoSchedule
          orchestratorVersion: 1.22.11
          osType: Linux
          tags:
            check2: value2
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2022-07-01"
        location: centralindia
        name: np102
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: User
          nodeLabels:
            check3: value3
          nodeTaints:
          - check3=value3:NoSchedule
          orchestratorVersion: 1.22.11
          osType: Windows
          tags:
            check3: value3
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demoaks
  proxyconfig: {}
  systemComponentsPlacement:
    daemonSetOverride:
      tolerations:
      - operator: Exists
    nodeSelector:
      check2: value2
    tolerations:
    - effect: NoSchedule
      key: check2
      operator: Equal
      value: value2
  type: aks

Important

Customer has to explicitly set "enabled": false in the addonProfile when an AKS cluster provisioning with addon profiles did not successfully complete. Removing the addon section from the config file will NOT remove the addon from the cluster.

An illustrative example of the v3 cluster spec YAML file for AKS is shown below

apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  name: demo-cluster-aks
  project: demo
spec:
  blueprintConfig:
    name: minimal
  cloudCredentials: demo_aks_creds
  config:
    kind: aksClusterConfig
    metadata:
      name: demo-cluster-aks
    spec:
      managedCluster:
        apiVersion: "2022-07-01"
        identity:
          type: UserAssigned
          userAssignedIdentities:
            ? /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/phani-rg-central-india/providers/Microsoft.ManagedIdentity/userAssignedIdentities/phani-user-assigned-id1
            : {}
        location: centralindia
        properties:
          addonProfiles:
            azureKeyvaultSecretsProvider:
              config:
                enableSecretRotation: "false"
                rotationPollInterval: 2m
              enabled: false
            azurePolicy:
              enabled: true
            httpApplicationRouting:
              enabled: false
            omsAgent:
              config:
                logAnalyticsWorkspaceResourceID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourcegroups/defaultresourcegroup-cin/providers/microsoft.operationalinsights/workspaces/DefaultWorkspace-a2252eb2-7a25-432b-a5ec-e18eba6f26b1-CIN
              enabled: false
          apiServerAccessProfile:
            enablePrivateCluster: false
          dnsPrefix: shobhit-test-av3-dns
          kubernetesVersion: 1.24.9
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2022-07-01"
        location: centralindia
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 1
          mode: System
          nodeLabels:
            nodepool-os: linux
            nodepool-type: System
          orchestratorVersion: 1.24.9
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2022-07-01"
        location: centralindia
        name: np2
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 1
          mode: User
          nodeLabels:
            nodepool-os: linux
            nodepool-type: System
          orchestratorVersion: 1.24.9
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2022-07-01"
        location: centralindia
        name: np3
        properties:
          count: 2
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 1
          mode: User
          nodeLabels:
            nodepool-os: linux
            nodepool-type: System
          orchestratorVersion: 1.24.9
          osType: Linux
          tags:
            asdas: asdasadas
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo-rg
  sharing:
    enabled: true
    projects:
    - name: defaultproject
  type: aks

Create Nodepool

./rctl create nodepool aks -f node-spec.yaml

An illustrative example of the cluster spec YAML file for AKS to create a v1 method of nodepool is shown below

apiVersion: rafay.io/v1alpha1
kind: aksClusterConfig
metadata:
  name: demo-aks
spec:
  resourceGroupName: my-rg
  nodePools:
  - type: Microsoft.ContainerService/managedClusters/agentPools
    apiVersion: "2021-05-01"
    name: agentpooly
    properties:
      count: 1
      maxPods: 30
      mode: System
      vmSize: Standard_B2s
  - type: Microsoft.ContainerService/managedClusters/agentPools
    apiVersion: "2021-05-01"
    name: agentpoolz
    properties:
      count: 1
      vmSize: Standard_B2s

Download Cluster Config

Use the below command to download the AKS v1 cluster config file

./rctl get cluster config <ClusterName> <ClusterConfigFileName.yaml>

Example:

/rctl get cluster config demo-aks demo-akscluster-config.yaml

To download a v3 cluster config, use the below command

./rctl get cluster config <cluster-name> --v3

Important

Download the cluster configuration only after the cluster is completely provisioned

List Clusters

To retrieve a specific AKS cluster, use the below command

./rctl get cluster <akscluster_name>

Output

./rctl get cluster demo-akscluster
+------------------------+-----------+-----------+---------------------------+
| NAME                   | TYPE      | OWNERSHIP | PROVISION STATUS          |
+------------------------+-----------+-----------+---------------------------+
| demo-akscluster        | azure-aks | self      | INFRA_CREATION_INPROGRESS |
+------------------------+-----------+-----------+---------------------------+

To retrieve a specific v3 cluster details, use the below command

./rctl get cluster demo-akscluster --v3

Example

./rctl get cluster demo-akscluster --v3
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+
| NAME                   | CREATED AT                    | OWNERSHIP | TYPE     | BLUEPRINT | PROVISION STATUS          |
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+
| demo-akscluster        | 2023-06-05 10:54:08 +0000 UTC | self      | aks      | minimal   | INFRA_CREATION_INPROGRESS |
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+

To view the entire v3 cluster config spec, use the below command

./rctl get cluster <akscluster_name> --v3 -o json

(or)

./rctl get cluster <akscluster_name> --v3 -o yaml

Delete Cluster

Users can delete one or more clusters with a single command

./rctl delete cluster <cluster-name>

(or)

./rctl delete cluster <cluster1-name> <cluster2-name>

Imperative

Create Cluster

Use this command to create an AKS cluster object in the configured project on the controller. You can optionally also specify the cluster blueprint during this step.

To create an AKS cluster: cloud credential, resource group and Kubernetes version are mandatory

./rctl create cluster aks <demo-aks> --cloud-creds <user-credentials> --resource-group <demo-resoursegroup> --version <k8s-version>

Users can add more flags based on the cluster requirement. The below help flag allows the users to view and utilize the required flags that can be associated with a new cluster

./rctl create cluster aks --help

Below is an example to create an AKS cluster with cluster autoscaler enabled and a custom blueprint:

./rctl create cluster aks <demo-aks> --cloud-creds <user-credentials> --resource-group <demo-resoursegroup> --version <k8s-version> --count 1 --enable-auto-scaling --min-count 1 --max-count 3 --blueprint <blueprint-name>

Create nodePool

To create a nodepool on AKS clusters, use the below command

./rctl create nodepool aks <node-pool-name> <cluster-name>

Users can add more flags based on the nodepool requirement. The below help flag allows the users to view and utilize the required flags that can be associated with a new nodepool

./rctl create nodepool aks --help

Below is an example to create a Nodepool on an AKS cluster with cluster autoscaler enabled:

./rctl create node-pool aks newpool ap-aks-2 --enable-auto-scaling --count 1 --max-pods-per-node 50 --node-labels pool=new

Update Nodepool

Use the below commands to update a nodePool

./rctl update node-pool aks <np-name> <cluster-name>
Users can add more flags based on the requirement to update different nodepool parameters. The below help flag allows the users to view the list of flags and utilize the required flags to update the nodepool

./rctl update node-pool aks --help
Currently, scaling (auto or manual) and mode change are supported for nodepools

Scaling command

Auto Scaling

./rctl update node-pool aks <np-name> <cluster-name> --scaling-type auto --min-count <n> --max-count <n>

Example:

./rctl update node-pool aks demo-np demo-cluster --scaling-type auto --min-count 1 --max-count 2

Manual Scaling

./rctl update node-pool aks <np-name> <cluster-name> --scaling-type manual --count <n>

Example:

./rctl update node-pool aks demo-np demo-cluster --scaling-type manual --count 3

Mode change command

./rctl update node-pool aks <np-name> <cluster-name> --mode

Delete Nodepool

Use the below command to delete a nodePool

./rctl delete node-pool <node-name> <cluster-name>

Cluster Upgrade

Use the below command to upgrade a cluster

./rctl upgrade cluster <cluster-name> --version <version>

Example:

./rctl upgrade cluster kal-aks-nov16 --version 1.20.9

Cluster Node Pool Upgrade

Users are allowed to upgrade the Cluster Node Pool with three modes

  1. Upgrade only node pools
./rctl upgrade cluster ClusterName --mode node-pools-only --node-pools <PoolName> --version <version>

(or)

./rctl upgrade node-pool <nodepool-name> <cluster-name> --version <version>
  1. Upgrade control plane only
./rctl upgrade cluster <cluster-name> --mode control-plane-only --version <version>
  1. Upgrade both node pools and control plane
./rctl upgrade cluster <cluster-name> --mode both --version <version>

Node Pool Image Upgrade

Use the below command to upgrade a Node Pool image

./rctl upgrade node-pool <nodepool-name> <cluster-name> --image-update