Secondary CIDR with VPC

As customers deploy more and more workloads to their Kubernetes clusters they can quickly get into situations where pods are consuming thousands of IP Addresses. This can quickly exhaust the pool of IP Addresses allocated to the primary CIDR block. In scenarios where it's not possible to recreate or extend the VPC's CIDR blocks they can deploy their worker nodes, pod network to a newly created larger CIDR block (e.g. 100.64.0.0/10) within the VPC. Newly created clusters can be configured to use the subnets defined in the secondary CIDR block. Existing clusters can migrate workloads over to the secondary CIDR block. This can be done by creating a new node group that utilize the subnets within the secondary CIDR block, once this is complete you can drain and delete the existing node group.

---

Architecture

In the described scenario it is possible to run the instances and pod network in subnets defined from the secondary CIDR block. The worker nodes primary and secondary Elastic Network Interfaces will run on subnets comprised of IP addresses defined in the secondary CIDR block. This allows the Kubernetes cluster the ability to utilize all the IP addresses allocated to all ENIs. Those IP Addresses previously reserved for the host network can now be used by the pod network increasing the node's pod density. Since the worker nodes ENIs are on subnets that are not routable they'll require access to a NAT Gateway to access external resources such as the internet.

---

VPC Configuration

Use the following VPC configuration as a reference to the content below.

Name	Subnet-Id	CIDR	AZ
SecondarySubnetPrivateUSWEST2A	subnet-08a7a7fdd3f23f18d	100.64.0.0/20	us-west-2a
SecondarySubnetPrivateUSWEST2B	subnet-09629aeeae61cfd36	100.64.16.0/20	us-west-2b
SecondarySubnetPrivateUSWEST2D	subnet-061adebec170d8ba3	100.64.32.0/20	us-west-2d
PrimarySubnetPrivateUSWEST2A	subnet-04e3ccd47c9ef9978	10.0.96.0/19	us-west-2a
PrimarySubnetPrivateUSWEST2B	subnet-06e2b1a5cc98cac91	10.0.120.0/19	us-west-2b
PrimarySubnetPrivateUSWEST2D	subnet-030531e64a35bac1a	10.0.160.0/19	us-west-2d
PrimarySubnetPublicUSWEST2A	subnet-016ec9f5fa892cac8	10.0.0.0/19	us-west-2a
PrimarySubnetPublicUSWEST2B	subnet-02aeb278cfa1d7835	10.0.32.0/19	us-west-2b
PrimarySubnetPublicUSWEST2D	subnet-001748a513e7d39e5	10.0.64.0/19	us-west-2d

---

Cluster Configuration

To utilize the subnets on the primary CIDR block for your control plane specify the primary subnets under the cluster settings during the cluster configuration as shown below.

Node Group Configuration

To utilize the subnets on the secondary CIDR block for your worker nodes and pod network specify the secondary subnets under the node group settings during the cluster configuration as shown below.

Sample Cluster Specification

If creating a cluster from a cluster specification you can define the subnets from the secondary CIDR block to be used for the node group under the subnets section in the node group as shown below.

kind: Cluster
metadata:
  name: my-eks-cluster
  project: my-project
spec:
  blueprint: default
  cloudprovider: my-cloud-provider
  clusterConfig: null
  cniprovider: aws-cni
  proxyconfig: {}
  type: eks
---
apiVersion: rafay.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  iam:
    withAddonPolicies:
      albIngress: true
      autoScaler: true
      externalDNS: true
  instanceType: t3.large
  maxSize: 2
  minSize: 2
  name: my-ng-on-secondary-cidr
  privateNetworking: true
  subnets:
  - subnet-08a7a7fdd3f23f18d
  - subnet-09629aeeae61cfd36
  - subnet-061adebec170d8ba3
  volumeSize: 80
  volumeType: gp3
metadata:
  name: my-eks-cluster
  region: us-west-2
  version: "1.21"
vpc:
  clusterEndpoints:
    privateAccess: true
    publicAccess: false
  nat:
    gateway: Single
  id: vpc-09d7879cdaea507e9
  subnets:
    private:
      subnet-04e3ccd47c9ef9978:
        id: subnet-04e3ccd47c9ef9978
      subnet-06e2b1a5cc98cac91:
        id: subnet-06e2b1a5cc98cac91
      subnet-030531e64a35bac1a:
        id: subnet-030531e64a35bac1a
    public:
      subnet-02aeb278cfa1d7835:
        id: subnet-02aeb278cfa1d7835
      subnet-016ec9f5fa892cac8:
        id: subnet-016ec9f5fa892cac8
      subnet-001748a513e7d39e5:
        id: subnet-001748a513e7d39e5