Overview
Azure Kubernetes Service (AKS) is a fully managed Kubernetes service provided by Microsoft Azure. We have developed an integration with AKS to ensure that users can provision AKS Clusters in any region using the RCTL CLI.
Declarative¶
You can also create the cluster based on a version controlled cluster spec that you can store in a Git repository. This enables users to develop automation for reproducible infrastructure.
Create Cluster¶
./rctl create cluster aks -f cluster-spec.yaml
Use the below v1 Spec to create a AKS Cluster
apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
name: demo-akscluster
project: default-project
spec:
blueprint: minimal
cloudprovider: cp_azure
clusterConfig:
apiVersion: rafay.io/v1alpha1
kind: aksClusterConfig
metadata:
name: demo-akscluster
spec:
managedCluster:
additionalMetadata:
acrProfile:
registries:
- acrName: demo-acr2
resourceGroupName: demo-rg
apiVersion: "2022-07-01"
identity:
type: UserAssigned
userAssignedIdentities:
? /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-cluster-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-cluster-identity
: {}
location: centralindia
properties:
aadProfile:
adminGroupObjectIDs:
- 9063e71a-fc36-4281-ac4b-be26189fa004
managed: true
apiServerAccessProfile:
enablePrivateCluster: true
dnsPrefix: aks-ui-01-dns
kubernetesVersion: 1.23.8
powerState:
code: Running
disableLocalAccounts: true
dnsPrefix: demo-dns
enableRBAC: true
identityProfile:
kubeletIdentity:
resourceId: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-cluster-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-kubelet-identity
kubernetesVersion: 1.27.7
networkProfile:
dnsServiceIP: 10.0.0.10
dockerBridgeCidr: 172.17.0.1/16
loadBalancerSku: standard
networkPlugin: azure
serviceCidr: 10.0.0.0/16
powerState:
code: Running
sku:
name: Basic
tier: Free
type: Microsoft.ContainerService/managedClusters
nodePools:
- apiVersion: "2022-07-01"
location: centralindia
name: primary
properties:
count: 2
enableAutoScaling: true
maxCount: 2
maxPods: 110
minCount: 2
mode: System
orchestratorVersion: 1.27.7
osType: Linux
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
vnetSubnetID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-fnf-nw-rg/providers/Microsoft.Network/virtualNetworks/demo-fnf-vnet/subnets/default
type: Microsoft.ContainerService/managedClusters/agentPools
resourceGroupName: demo-rg
type: aks
Below is an example of the AKS addon yaml file
apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
name: demo-aks-addon
project: defaultproject
spec:
blueprint: default-aks
cloudprovider: azure-credentials
clusterConfig:
apiVersion: rafay.io/v1alpha1
kind: aksClusterConfig
metadata:
name: demo-aks-addon
spec:
managedCluster:
apiVersion: "2022-07-01"
identity:
type: SystemAssigned
location: centralindia
properties:
addonProfiles:
azureKeyVaultSecretsProvider:
config:
enableSecretRotation: "true"
rotationPollInterval: 1m
enabled: true
azurePolicy:
enabled: true
httpApplicationRouting:
enabled: true
ingressApplicationGateway:
config:
applicationGatewayId: "Application Gateway ID"
enabled: true
omsAgent:
config:
logAnalyticsWorkspaceResourceID: "Resource ID"
enabled: true
openServiceMesh:
enabled: true
apiServerAccessProfile:
enablePrivateCluster: false
dnsPrefix: demo-aks-addon-dns
kubernetesVersion: 1.22.11
networkProfile:
dnsServiceIP: 10.0.0.10
dockerBridgeCidr: 172.17.0.1/16
loadBalancerSku: standard
networkPlugin: azure
networkPolicy: azure
serviceCidr: 10.0.0.0/16
sku:
name: Basic
tier: Free
type: Microsoft.ContainerService/managedClusters
nodePools:
- apiVersion: "2022-07-01"
location: centralindia
name: primary
properties:
count: 1
enableAutoScaling: true
maxCount: 1
maxPods: 40
minCount: 1
mode: System
nodeLabels:
check1: value1
nodeTaints:
- check1=value1:PreferNoSchedule
orchestratorVersion: 1.22.11
osType: Linux
tags:
check1: value1
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
- apiVersion: "2022-07-01"
location: centralindia
name: np1
properties:
count: 1
enableAutoScaling: true
maxCount: 1
maxPods: 40
minCount: 1
mode: User
nodeLabels:
check2: value2
nodeTaints:
- check2=value2:NoSchedule
orchestratorVersion: 1.22.11
osType: Linux
tags:
check2: value2
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
- apiVersion: "2022-07-01"
location: centralindia
name: np102
properties:
count: 1
enableAutoScaling: true
maxCount: 1
maxPods: 40
minCount: 1
mode: User
nodeLabels:
check3: value3
nodeTaints:
- check3=value3:NoSchedule
orchestratorVersion: 1.22.11
osType: Windows
tags:
check3: value3
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
resourceGroupName: demoaks
proxyconfig: {}
systemComponentsPlacement:
daemonSetOverride:
tolerations:
- operator: Exists
nodeSelector:
check2: value2
tolerations:
- effect: NoSchedule
key: check2
operator: Equal
value: value2
type: aks
Important
Customer has to explicitly set "enabled": false in the addonProfile when an AKS cluster provisioning with addon profiles did not successfully complete. Removing the addon section from the config file will NOT remove the addon from the cluster.
An illustrative example of the v3 cluster spec YAML file for AKS is shown below
apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
name: demo-cluster-aks
project: demo
spec:
blueprintConfig:
name: minimal
cloudCredentials: demo_aks_creds
config:
kind: aksClusterConfig
metadata:
name: demo-cluster-aks
spec:
managedCluster:
apiVersion: "2022-07-01"
identity:
type: UserAssigned
userAssignedIdentities:
? /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/phani-rg-central-india/providers/Microsoft.ManagedIdentity/userAssignedIdentities/phani-user-assigned-id1
: {}
location: centralindia
properties:
addonProfiles:
azureKeyvaultSecretsProvider:
config:
enableSecretRotation: "false"
rotationPollInterval: 2m
enabled: false
azurePolicy:
enabled: true
httpApplicationRouting:
enabled: false
omsAgent:
config:
logAnalyticsWorkspaceResourceID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourcegroups/defaultresourcegroup-cin/providers/microsoft.operationalinsights/workspaces/DefaultWorkspace-a2252eb2-7a25-432b-a5ec-e18eba6f26b1-CIN
enabled: false
apiServerAccessProfile:
enablePrivateCluster: false
dnsPrefix: demo-test-av3-dns
kubernetesVersion: 1.24.9
networkProfile:
loadBalancerSku: standard
networkPlugin: kubenet
sku:
name: Basic
tier: Free
type: Microsoft.ContainerService/managedClusters
nodePools:
- apiVersion: "2022-07-01"
location: centralindia
name: primary
properties:
count: 1
enableAutoScaling: true
maxCount: 2
maxPods: 40
minCount: 1
mode: System
nodeLabels:
nodepool-os: linux
nodepool-type: System
orchestratorVersion: 1.24.9
osType: Linux
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
- apiVersion: "2022-07-01"
location: centralindia
name: np2
properties:
count: 1
enableAutoScaling: true
maxCount: 2
maxPods: 40
minCount: 1
mode: User
nodeLabels:
nodepool-os: linux
nodepool-type: System
orchestratorVersion: 1.24.9
osType: Linux
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
- apiVersion: "2022-07-01"
location: centralindia
name: np3
properties:
count: 2
enableAutoScaling: true
maxCount: 2
maxPods: 40
minCount: 1
mode: User
nodeLabels:
nodepool-os: linux
nodepool-type: System
orchestratorVersion: 1.24.9
osType: Linux
tags:
asdas: asdasadas
type: VirtualMachineScaleSets
vmSize: Standard_B4ms
type: Microsoft.ContainerService/managedClusters/agentPools
resourceGroupName: demo-rg
sharing:
enabled: true
projects:
- name: defaultproject
type: aks
Create Nodepool¶
./rctl create nodepool aks -f node-spec.yaml
An illustrative example of the cluster spec YAML file for AKS to create a v1 method of nodepool is shown below
apiVersion: rafay.io/v1alpha1
kind: aksClusterConfig
metadata:
name: demo-aks
spec:
resourceGroupName: my-rg
nodePools:
- type: Microsoft.ContainerService/managedClusters/agentPools
apiVersion: "2021-05-01"
name: agentpooly
properties:
count: 1
maxPods: 30
mode: System
vmSize: Standard_B2s
- type: Microsoft.ContainerService/managedClusters/agentPools
apiVersion: "2021-05-01"
name: agentpoolz
properties:
count: 1
vmSize: Standard_B2s
Download Cluster Config¶
Use the below command to download the AKS v1 cluster config file
./rctl get cluster config <ClusterName> <ClusterConfigFileName.yaml>
Example:
/rctl get cluster config demo-aks demo-akscluster-config.yaml
To download a v3 cluster config, use the below command
./rctl get cluster config <cluster-name> --v3
Important
Download the cluster configuration only after the cluster is completely provisioned
List Clusters¶
To retrieve a specific AKS cluster, use the below command
./rctl get cluster <akscluster_name>
Output
./rctl get cluster demo-akscluster
+------------------------+-----------+-----------+---------------------------+
| NAME | TYPE | OWNERSHIP | PROVISION STATUS |
+------------------------+-----------+-----------+---------------------------+
| demo-akscluster | azure-aks | self | INFRA_CREATION_INPROGRESS |
+------------------------+-----------+-----------+---------------------------+
To retrieve a specific v3 cluster details, use the below command
./rctl get cluster demo-akscluster --v3
Example
./rctl get cluster demo-akscluster --v3
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+
| NAME | CREATED AT | OWNERSHIP | TYPE | BLUEPRINT | PROVISION STATUS |
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+
| demo-akscluster | 2023-06-05 10:54:08 +0000 UTC | self | aks | minimal | INFRA_CREATION_INPROGRESS |
+------------------------+-------------------------------+-----------+----------+-----------+---------------------------+
To view the entire v3 cluster config spec, use the below command
./rctl get cluster <akscluster_name> --v3 -o json
(or)
./rctl get cluster <akscluster_name> --v3 -o yaml
Delete Cluster¶
Users can delete one or more clusters with a single command
./rctl delete cluster <cluster-name>
(or)
./rctl delete cluster <cluster1-name> <cluster2-name>
Imperative¶
Create Cluster¶
Use this command to create an AKS cluster object in the configured project on the controller. You can optionally also specify the cluster blueprint during this step.
To create an AKS cluster: cloud credential, resource group and Kubernetes version are mandatory
./rctl create cluster aks <demo-aks> --cloud-creds <user-credentials> --resource-group <demo-resoursegroup> --version <k8s-version>
Users can add more flags based on the cluster requirement. The below help flag allows the users to view and utilize the required flags that can be associated with a new cluster
./rctl create cluster aks --help
Below is an example to create an AKS cluster with cluster autoscaler enabled and a custom blueprint:
./rctl create cluster aks <demo-aks> --cloud-creds <user-credentials> --resource-group <demo-resoursegroup> --version <k8s-version> --count 1 --enable-auto-scaling --min-count 1 --max-count 3 --blueprint <blueprint-name>
Create nodePool¶
To create a nodepool on AKS clusters, use the below command
./rctl create nodepool aks <node-pool-name> <cluster-name>
Users can add more flags based on the nodepool requirement. The below help flag allows the users to view and utilize the required flags that can be associated with a new nodepool
./rctl create nodepool aks --help
Below is an example to create a Nodepool on an AKS cluster with cluster autoscaler enabled:
./rctl create node-pool aks newpool ap-aks-2 --enable-auto-scaling --count 1 --max-pods-per-node 50 --node-labels pool=new
Update Nodepool¶
Use the below commands to update a nodePool
./rctl update node-pool aks <np-name> <cluster-name>
./rctl update node-pool aks --help
Scaling command
Auto Scaling
./rctl update node-pool aks <np-name> <cluster-name> --scaling-type auto --min-count <n> --max-count <n>
Example:
./rctl update node-pool aks demo-np demo-cluster --scaling-type auto --min-count 1 --max-count 2
Manual Scaling
./rctl update node-pool aks <np-name> <cluster-name> --scaling-type manual --count <n>
Example:
./rctl update node-pool aks demo-np demo-cluster --scaling-type manual --count 3
Mode change command
./rctl update node-pool aks <np-name> <cluster-name> --mode
Delete Nodepool¶
Use the below command to delete a nodePool
./rctl delete node-pool <node-name> <cluster-name>
Cluster Upgrade¶
Use the below command to upgrade a cluster
./rctl upgrade cluster <cluster-name> --version <version>
Example:
./rctl upgrade cluster kal-aks-nov16 --version 1.20.9
Cluster Node Pool Upgrade¶
Users are allowed to upgrade the Cluster Node Pool with three modes
- Upgrade only node pools
./rctl upgrade cluster ClusterName --mode node-pools-only --node-pools <PoolName> --version <version>
(or)
./rctl upgrade node-pool <nodepool-name> <cluster-name> --version <version>
- Upgrade control plane only
./rctl upgrade cluster <cluster-name> --mode control-plane-only --version <version>
- Upgrade both node pools and control plane
./rctl upgrade cluster <cluster-name> --mode both --version <version>
Node Pool Image Upgrade¶
Use the below command to upgrade a Node Pool image
./rctl upgrade node-pool <nodepool-name> <cluster-name> --image-update
Start/Stop Cluster¶
To stop a running cluster, use the below command
./rctl start cluster <cluster_name>
To start a stopped cluster, use the below command
./rctl stop cluster <cluster_name>