preBootstrapCommands

preBootstrapCommands are used to prepare nodes and run before a cluster is brought up. An example are any OS level configuration changes that have to be done for the cluster to come up successfully, like inserting CA certs or changes to containerd.

preBootstrapCommands is a list of commands (one or more) to run on a target cluster. These commands are input using the cluster configuration.

Notes

• This setting is optional.
• preBootstrapCommands will run every time a node comes up, during cluster creation and cluster or node pool scaling. Examples: Node restart and node creation.
• preBootstrapCommands are only executed on Linux-based cluster nodes.

---

Example Use Case

If a cluster and its nodes need to be behind a proxy because of enterprise security guidelines. The nodes need to have a CA cert injected for the nodes to be able to communicate outside. In this use case, if the user configures the preBootstrapCommands at the time of configuring their cluster, the provisioning process will read the user's configuration and inject the CA cert while the node is coming up.

Configure preBootstrapCommands

Using the Console

When provisioning a GKE cluster from the Console, add the preBootstrapCommands under Advanced Settings.

Using the CLI

When provisioning a GKE cluster using the CLI, add the preBootstrapCommands in the configuration file as shown in the specification below.

apiVersion: infra.k8smgmt.io/v2
kind: Cluster
metadata:
  annotations:
    rafay.dev/description: ""
  name: test-tf-qc
  project: demo-p1
spec:
  blueprint:
    name: demo-bp1
    version: ver0
  cloudCredentials: demo-qa
  config:
    controlPlaneVersion: "1.28"
    feature:
      cloudLoggingComponents:
      - SYSTEM_COMPONENTS
      - WORKLOADS
      cloudMonitoringComponents:
      - SYSTEM_COMPONENTS
      enableCloudLogging: true
      enableCloudMonitoring: true
      enableComputeEnginePersistentDiskCSIDriver: true
    location:
      type: zonal
      zone: us-central1-c
    name: test-tf-qc
    network:
      enableVPCNativeTraffic: true
      maxPodsPerNode: 110
      name: demo-vpc1
      networkAccess:
        privacy: private
        privateCluster:
          controlPlaneIPRange: 172.16.1.0/28
          disableSNAT: true
          enableAccessControlPlaneExternalIP: true
          enableAccessControlPlaneGlobal: true
      nodeSubnetName: demo-vpc1-sub1
    nodePools:
    - machineConfig:
        accelerators:
        - acceleratorSharing:
            maxSharedClients: 2
            strategy: TIME_SHARING
          count: 1
          driverInstallation:
            googleManagedDriverInstallation:
              version: LATEST
            type: google-managed
          type: nvidia-tesla-t4
        bootDiskSize: 100
        bootDiskType: pd-standard
        imageType: COS_CONTAINERD
        machineType: n1-standard-4
        reservationAffinity:
          consumeReservationType: any
      name: demo-dnp
      nodeVersion: "1.28"
      size: 1
    - machineConfig:
        bootDiskSize: 100
        bootDiskType: pd-standard
        imageType: COS_CONTAINERD
        machineType: e2-standard-4
        reservationAffinity: {}
      name: demo-nodepool2
      nodeVersion: "1.26"
      size: 1
    preBootstrapCommands:
    - |-
      mkdir -p ${ROOT_DIR}/tmp/demo && echo -e "hello
      world - May23rd 2023" >> ${ROOT_DIR}/tmp/demo/test_file.txt
    project: defaultproject
  type: Gke

See GKE Config Schema for more information about the configuration file.

preBootstrapCommands Guidelines

• To refer to the root directory for a node, use the ${ROOT_DIR} variable. For example, /etc/ssl/certs/ should be ${ROOT_DIR}/etc/ssl/certs for the preBootstrapCommands.
• To detect the container runtime, use the IS_CONTAINERD and IS_DOCKER variables. Both are boolean variables. The IS_CONTAINERD variable is set to true when Kubelet is using containerd as a container runtime.
• To change the root directory for a specific command, prefix the command with chroot "${ROOT_DIR}". For example, chroot "${ROOT_DIR}" systemctl restart containerd.
• base64 encoding the command is not required. Users can use base64 encoding to avoid any issues with any special characters that might be in the command. Since the commands are run as-is, the commands must be written in a way to be decoded upon execution.

preBootstrapCommands Examples

Most Linux commands are valid for preBootstrapCommands.

Download Kubectl

curl -L "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" -o /tmp/kubectl
mv /tmp/kubectl /usr/local/bin/ && chmod +x /usr/local/bin/kubectl

Enable IP Forwarding

sysctl -w net.ipv4.ip_forward=1
sed -i 's/net.ipv4.ip_forward=0/net.ipv4.ip_forward=1/' /etc/sysctl.conf
sysctl --quiet --system

preBootstrapCommands cannot pass bash-script. A work-around is to base64 encode the script, then convert it to a command that will decode and execute.

echo "<base64-encoded-script>" | base64 -d | bash

Invalid Example

<something with /mnt/ >

preBootstrapCommands Execution Flow

The preBootstrapCommands run before applying Cluster Operator Spec.

sequenceDiagram
    Rafay Core->>Bootstrap VM: Send kubectl commands to the infra-agent
    Note over Bootstrap VM: Infra-Agent
    Bootstrap VM->>Target GKE Cluster: Creates the DaemonSet and ConfigMap
    Note over Target GKE Cluster: DaemonSet
    Note over Target GKE Cluster: ConfigMap

DaemonSet is used to execute the preBootstrapCommands. This happens in two steps:

• Setup - This sets up and runs the preBootstrapCommands on the target cluster. The preBootstrapCommands come from the user's configuration input.
• Validation - After the setup is complete, the controller looks for any failures in the preBootstrapCommands execution on the nodes. Any errors should appear in the console.

Troubleshooting

The preBootstrapCommands are executed on all nodes spun up at the time of cluster provisioning.

Errors Display in Console

If any of the preBootstrapCommands fail, it is automatically reflected on the cluster provisioning page, with the pod name and error line.

Errors in Pod Logs

If all preBootstrapCommands are executed successfully on all nodes, then the cluster provisioning process proceeds to the next step.

Once the cluster is provisioned, you can track the preBootstrapCommands output using the kubectl command kubectl get pods -o wide.

If CrashLoopBackoff displays for the pod status, check the pod log file for more information.

Example Pod Log File Error

---