Aug

v2.8 - Self Hosted

23 Aug, 2024

This release introduces:

• Fresh Controller Installation:
• A new controller installation is available for EKS, GKE, and Airgapped Controller environments.
• Upgrade Path for 2.7 to 2.8:
  • An upgrade path is provided from version 2.7 to 2.8 for GKE, EKS, and Airgapped controllers.
• Config Builder Tool:
  • The 2.8 controller package includes a convenient config builder tool.
  • This command-line tool simplifies configuration by prompting users for input and generating YAML files for Baremetal, EKS, and GKE environments.
  • Usage:

    ./configbuilder baremetal
    

• Rcollect Enhancements The Rcollect tool has been enhanced with the following improvements:

  • Background Execution: Rcollect can now be run in the background as a terminal process.

    ./rcollect kuberesources --background --timeout 10m --since 10h
    

  • Minion and Master Version Collection: Rcollect now collects minion and master versions as part of the log collection process.
  • Timeout Flag: A new timeout flag allows you to specify a duration for Rcollect to run.

---

Amazon EKS

Enabling secret encryption on an existing cluster

Currently, secret encryption with a KMS key can only be configured during cluster creation (Day 0) using the 'save and customize' option. Support is now added to enable secret encryption on existing clusters (Day 2) as well, along with the option to configure it during cluster creation (Day 0) using the cluster form.

For more information on this feature, please refer here.

Day 0 Configuration

For new clusters, you can find this configuration in the cluster settings. Based on valid cloud credentials, you will see the option to enable secret encryption during Day 0 configuration.

Day 2 Configuration

For existing clusters, you can navigate to the cluster view, then go to Configuration where you will find an option to enable Secret Encryption.

RCTL Cluster Configuration with Secret Encryption

kind: Cluster
metadata:
  name: cluster-config
  project: defaultproject
spec:
  blueprint: minimal
  blueprintversion: 2.7.0
  cloudprovider: eks-cloud
  cniprovider: aws-cni
  proxyconfig: {}
  type: eks
---
addons:
- name: coredns
  version: v1.10.1-eksbuild.4
- name: vpc-cni
  version: v1.15.1-eksbuild.1
- name: kube-proxy
  version: v1.28.2-eksbuild.2
- name: aws-ebs-csi-driver
  version: latest
apiVersion: rafay.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  iam:
    withAddonPolicies:
      autoScaler: true
  instanceTypes:
  - t3.xlarge
  maxSize: 2
  minSize: 0
  name: ng-b476082a
  version: "1.28"
  volumeSize: 80
  volumeType: gp3
metadata:
  name: cluster-config
  region: us-west-2
  tags:
    email: demo@rafay.co
    env: dev
  version: "1.28"
secretsEncryption:
  keyARN: arn:aws:kms:us-west-2:xxxxxxxxxxxxx:key/xxxxxxxxxxxxxxx
vpc:
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: true
    publicAccess: false
  nat:
    gateway: Single

IAM Permissions Required

When enabling secret encryption with a KMS key on EKS clusters, ensure that the IAM roles or users performing these actions have the following permissions assigned:

• For key listing: kms:ListKeys
• For secret encryption: kms:DescribeKey, kms:CreateGrant
• For Encryption on Existing EKS Cluster : eks:AssociateEncryptionConfig

These permissions are necessary for listing KMS keys, managing encryption grants, and associating an encryption configuration to an existing cluster that don't already have encryption enabled.

Irreversible Action: Secrets Encryption

Once enabled, secrets encryption cannot be disabled. This action is irreversible.

Amazon Linux 2023 Support

This release introduces support for AmazonLinux2023 in node groups. Users can now leverage the benefits of AL2023 by creating node groups to use this Amazon Linux version.

Node Group Configuration with AL2023

Example: Creating an EKS Node Group with AL2023 based Custom AMI

managedNodeGroups:
- ami: <custom_ami>
  desiredCapacity: 1
  iam:
    withAddonPolicies:
      autoScaler: true
  instanceTypes:
  - t3.xlarge
  maxSize: 1
  minSize: 1
  name: al2023_customami
  overrideBootstrapCommand: |
    [AmazonLinux2023]
  ssh:
    allow: true
    publicKeyName: <awskey>
  volumeSize: 80
  volumeType: gp3

Amazon EKS Cluster with AL2023 node

---

Amazon EKS and Azure AKS

Toleration Support for FleetOps Pre-Hook/Post-Hook Configuration

In this release, we are adding toleration support in the FleetOps pre-hook and post-hook configuration. This enhancement allows users to specify tolerations for hooks so that they can get scheduled on specific nodes and avoid being scheduled on nodes where they are not allowed.

Supported Interfaces

• UI
• RCTL
• SystemSync
• Swagger API's

Note: Support for Terraform will be delivered in a subsequent release. Additionally, Node Selectors and Tolerations are applicable only when the Runner type is set to Cluster.

For more information on this feature, please refer to Fleet Tolerations here

---

GKE Clusters

Network Policy and Dataplane V2 for GKE Clusters

This enhancement provides users with advanced networking capabilities ensuring improved security and performance for their applications. Dataplane V2 leverages the power of eBPF, providing enhanced observability, scalability, and resilience, enabling seamless traffic management across clusters. Additionally, Network Policy support allows fine-grained control over network traffic, ensuring that only authorized communications occur between services.

For more information on this feature, please refer here.

---

Upstream Kubernetes for Bare Metal and VMs

GitOps SystemSync with Write Back to Git

With this feature, users will be able to configure the platform to continuously sync cluster specifications for upstream Kubernetes clusters with a Git repository. Changes can be made in a bidirectional manner.

For more information on this, please refer here.

• If the cluster specification is updated in the Git repository, the platform will update the corresponding upstream Kubernetes cluster to bring it to the desired state
• If the upstream Kubernetes cluster's state is modified by an authorized user using the UI or CLI, the changes are automatically written back to the configured Git repository
• To enable GitOps SystemSync for Upstream Kubernetes (MKS) on Bare Metal and VMs, users must create cloud credentials. These credentials are necessary for GitOps Agent to facilitate the GitOps synchronization.

GitOps Agent Update Required

To use GitOps for upstream cluster types, you must update the GitOps agent to version r2.8.0+.

Cloud Credential Support for Upstream Kubernetes Cluster(MKS)

This release introduces support for managing cloud credentials for upstream Kubernetes clusters (MKS) within the platform. These credentials are essential for enabling GitOps SystemSync functionality and have been integrated into the UI, RCTL, and SystemSync interfaces.

SystemComponentsPlacement Support

SystemComponentsPlacement allows configuring the scheduling of system components on dedicated nodes. This release introduces support for systemComponentsPlacement as part of the new extended configuration schema. This functionality is currently supported in RCTL, V3 APIs and GitOps System Sync.

For more information on this, please refer here.

Using systemComponentsPlacement in RCTL

To utilize systemComponentsPlacement in RCTL, you need to pass the --v3 flag when applying your cluster configuration. Here's an example:

./rctl apply -f <cluster configuration> --v3

apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  name: demo-mks
  project: defaultproject
spec:
  blueprint:
    name: minimal
    version: latest
  cloudCredentials: demo-mks-creds
  config:
    autoApproveNodes: true
    dedicatedControlPlane: true
    highAvailability: true
    kubernetesVersion: v1.28.9
    location: sanjose-us
    network:
      cni:
        name: Calico
        version: 3.26.1
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    nodes:
    - arch: amd64
      hostname: demo-mks-scale-w-tb-2
      operatingSystem: Ubuntu20.04
      privateIP: 10.12.105.227
      roles:
      - Worker
    - arch: amd64
      hostname: demo-mks-scale-w-tb-7
      labels:
        app: infra
      operatingSystem: Ubuntu20.04
      privateIP: 10.12.110.50
      roles:
      - Worker
      taints:
      - effect: NoSchedule
        key: app
        value: infra
    - arch: amd64
      hostname: demo-mks-scale-w-tb-3
      operatingSystem: Ubuntu20.04
      privateIP: 10.12.29.164
      roles:
      - Worker
    - arch: amd64
      hostname: demo-mks-scale-w-tb-6
      labels:
        app: infra
      operatingSystem: Ubuntu20.04
      privateIP: 10.12.101.223
      roles:
      - Worker
      taints:
      - effect: NoSchedule
        key: app
        value: infra
    - arch: amd64
      hostname: demo-mks-scale-c-tb-3
      operatingSystem: Ubuntu20.04
      privateIP: 10.12.118.110
      roles:
      - ControlPlane
  systemComponentsPlacement:
    nodeSelector:
      app: infra
    tolerations:
    - effect: NoSchedule
      key: app
      operator: Equal
      value: infra
    - effect: NoSchedule
      key: infra
      operator: Equal
      value: rafay
  type: mks

Kubernetes Certificate Rotation

Support is now added to rotate Kubernetes certificates for upstream Kubernetes clusters via UI and API. This can be done either manually or automatically based on certificate expiry. This capability streamlines the entire certificate rotation process.

For more information on this feature, please refer here.

Note

Please note that Kubernetes certificate rotation feature described above is not supported in Windows-based upstream clusters.

CentOS 7 EOL

Due to CentOS 7 reaching its end-of-life (EOL) on June 30, 2024, this release no longer supports creating new clusters using CentOS 7.

We recommend transitioning to alternative supported operating systems like Rocky Linux, AlmaLinux, or RHEL as replacements for CentOS 7.

Read more about CentOS 7 End of Life.

---

Terraform Provider

Caching

There are certain scenarios where diff is shown even when there are no changes. A caching mechanism has been implemented for TF schema to alleviate this issue for the following resources:

• rafay_eks_cluster
• rafay_aks_cluster

Important

Limited Access - This capability is selectively enabled for customer orgs. Please reach out to support if you want to get this enabled for your organization. Support for more resources will be added with future releases.

Note

If users reorder elements in a list within the Terraform configuration, Terraform sees this as a difference during re-apply. This doesn't necessarily mean that the infrastructure needs updating, it indicates that the configuration has changed. Issues related to Terraform behavior like this are not resolved by caching.

---

Deprecation

Cluster Templates

The Cluster Templates feature set is deprecated with this release and support for this feature will be removed in a future release. This means that no enhancements or bug fixes will be added to this feature. Users can leverage Environment Manager to achieve the same use case.

vSphere LCM

The vSphere Cluster Lifecycle Management capability is deprecated with this release, and support for this feature will be removed in a future release. Users are encouraged to migrate to the Upstream Kubernetes for Bare Metal and VMs (MKS) offering. The infrastructure for Upstream Kubernetes on vSphere environment can be deployed leveraging Environment Manager.

---

Environment Manager

RCTL support

With the introduction of this capability, it is possible to execute Environment Manager workflows using the RCTL Command Line interface utility. RCTL CLI can be embedded into the preferred workflow automation (e.g. CI/CD pipeline) to perform operations such as creation of templates, deployment/shutting down environments.

For more information on this feature, please refer here.

Support for HCP Terraform Provider

Platform teams will be able to seamlessly integrate and leverage existing investments in HCP Terraform with the introduction of this provider option. With this integration, Platform teams can define templates and enable self-service for developers/data scientists within Rafay while the infrastructure provisioning & management of state files can be handled by HCP Terraform.

For more information on this, please refer here.

Note: This provider option is only available to HashiCorp licensed customers.

Support for OpenTofu Provider

Support for OpenTofu provider option is now added as part of the resource template configuration. This provides customers the flexibility to choose the provider of choice for infrastructure provisioning and state management.

For more information on this and steps for migrating from Terraform Provider to OpenTofu provider, please refer here.

Note: Support for Terraform provider option is being deprecated with the introduction of this option.

---

Clusters

Resources page

Sorting support is now available for all columns.

---

Secrets Management

Vault Integration

Vault Agent Injector version that is deployed to the clusters has been updated to v1.17.2 with this release.

---

2.8 Release Bug Fixes

Bug ID	Description
RC-35542	MKS: Containerd version is not being updated on all nodes during K8s upgrade
RC-34562	OpenCost agent is not deployed with a priorityClassName
RC-34492	AKS: Unable to update cloud credentials associated with a cluster using RCTL v3 spec
RC-32012	GKE: Cluster creation fails when the name includes rafay
RC-36073	Incorrect kubeconfig when downloaded through Terraform
RC-33947	Unintuitive error message on a blueprint sync failure when the add-on name is the same as the Rafay system add-on
RC-36557	UI: Network policies do not persist when creating a new blueprint version

---