Part 1: Provision

What Will You Do¶

In this part of the self-paced exercise, you will provision an Amazon EKS cluster based on a declarative cluster specification which contains two (2) managed nodes groups. The first managed node group will be comprised of on-demand compute instances and run the system level resources while the second managed node group will be comprised of spot instances and run application workloads. The cluster will use the minimal blueprint.

Step 1: Provision Cluster¶

In this step, we will create the declarative cluster specification file and use the RCTL CLI to provision the cluster from the specification file.

Save the below specification file to your computer as "eks-spot-cluster.yaml". Note, the highlighted sections in the spec will need to be updated to match your environment.

apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  # The name of the cluster
  name: eks-spot-cluster
  # The name of the project the cluster will be created in
  project: defaultproject
spec:
  blueprintConfig:
    # The name of the blueprint the cluster will use
    name: minimal
    # The version of the blueprint the cluster will use
    version: 
  # The name of the cloud credential that will be used to create the cluster 
  cloudCredentials: aws-cloud-credential
  config:
    # The EKS addons that will be applied to the cluster
    addons:
    - name: kube-proxy
      version: latest
    - name: vpc-cni
      version: latest
    - name: coredns
      version: latest
    managedNodeGroups:
      # The AWS AMI family type the nodes will use
    - amiFamily: AmazonLinux2
      # The desired number of nodes that can run in the node group 
      desiredCapacity: 2
      iam:
        withAddonPolicies:
          # Enables the IAM policy for cluster autoscaler
          autoScaler: true
      # The AWS EC2 instance types that will be used for the spot nodes
      instanceTypes:
      - t3.large
      - t2.large
      # The maximum number of nodes that can run in the node group
      maxSize: 4
      # The minimum number of nodes that can run in the node group
      minSize: 2
      # The name of the node group that will be created in AWS
      name: managed-spot
      # Enable the use of spot instances in the node group
      spot: true
      # The AWS AMI family type the nodes will use
    - amiFamily: AmazonLinux2
      # The desired number of nodes that can run in the node group
      desiredCapacity: 1
      iam:
        withAddonPolicies:
          # Enables the IAM policy for cluster autoscaler
          autoScaler: true
      # The AWS EC2 instance type that will be used for the nodes    
      instanceType: t3.large
      # The labels applied to the nodes in the node group
      labels:
        nodes: system
      # The maximum number of nodes that can run in the node group
      maxSize: 2
      # The minimum number of nodes that can run in the node group
      minSize: 1
      # The name of the node group that will be created in AWS
      name: managed-system
      # Apply taints to the node group to allow only system resources to be scheduled on these nodes
      taints:
      - effect: NoSchedule
        key: components
        value: system
    metadata:
      # The name of the cluster
      name: eks-spot-cluster
      # The AWS region the cluster will be created in
      region: us-west-2
      # The Kubernetes version that will be installed on the cluster 
      version: latest
    vpc:
      # AutoAllocateIPV6 requests an IPv6 CIDR block with /56 prefix for the VPC
      autoAllocateIPv6: false
      clusterEndpoints:
        # Enables private access to the Kubernetes API server endpoints
        privateAccess: true
        # Enables public access to the Kubernetes API server endpoints
        publicAccess: false
      # The CIDR that will be used  by the cluster VPC
      cidr: 192.168.0.0/16
  # Configure the scheduler to only place system resources on the managed-system node group 
  systemComponentsPlacement:
    nodeSelector:
      nodes: system
    tolerations:
    - effect: NoSchedule
      key: components
      operator: Equal
      value: system
  type: aws-eks

Update the following sections of the specification file with details to match your environment

Update the name section with the name of the cluster to be created and the project section with the name of the Rafay project you previously created
```
  name: eks-spot-cluster
  project: defaultproject
```
Update the cloudCredentials section with the name of the AWS cloud credential that was previously created
```
  cloudCredentials: aws-cloud-credential
```
Update the name and region sections with the cluster name and the AWS region where the cluster will be located
```
    metadata:
      name: eks-spot-cluster
      region: us-west-2
```
Save the updates that were made to the file
Open Terminal (on macOS/Linux) or Command Prompt (Windows) and navigate to the folder where you saved the file
Execute the following command to provision the cluster from the specification file previously saved
```
./rctl apply -f eks-spot-cluster.yaml
```
Login to the web console
Navigate to your project
Select Infrastructure -> Clusters

Provisioning in Process

Click on the cluster name to monitor progress

Provisioning in Process

Provisioning the infrastructure will take approximately 45 minutes to complete. The final step in the process is the blueprint sync.

Step 2: Verify Cluster¶

Once provisioning is complete, you should see the cluster in the web console

Provisioned Cluster

Click on the kubectl link and type the following command

kubectl get nodes

You should see something like the following

NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-34-232.us-west-2.compute.internal   Ready    <none>   8m46s   v1.23.13-eks-fb459a0
ip-192-168-4-152.us-west-2.compute.internal    Ready    <none>   8m10s   v1.23.13-eks-fb459a0
ip-192-168-69-188.us-west-2.compute.internal   Ready    <none>   8m9s    v1.23.13-eks-fb459a0

Now, we will verify that the one node group is using spot instances.

Select Infrastructure -> Clusters
Click on the cluster name
Click on the "Node Group" tab

You will see the "managed-spot" node group shows the use of spot instances and the spot instance types that can be used.

Spot Node Group

Next, we will verify the nodes in the node group are labeled to use spot instances.

Click on the "Nodes" tab
Click on "Overview" on one of the nodes in the "managed-spot" node group

You will see the capacity type is set to "SPOT"

Spot Nodes

Now we will verify the system resources are running on the on-demand node group.

Click "Nodes" in the tree at the top of the page to return to the nodes tab
Locate the "Node ID" of the node in the "managed-system" node group

Node ID

Click on the "Resources" tab
Click on "Pods" in the left side window
Select "rafay-system" from the namespace drop down menu
Click the gear icon on the right side of the page and select "Node"

You will see that all of the system components are running on the "managed-system" node that was previously identified.

System Components

Recap¶

Congratulations! At this point, you have successfully provisioned an Amazon EKS cluster with a managed spot instance node group in your AWS account using the RCTL CLI.