Skip to content

Troubleshooting

Cluster provisioning will fail if issues are detected and cannot be automatically overcome. When this occurs, the user is presented with an error message on the Web Console with a link to download the "error logs".

Provisioning Failure


Environmental

Environmental or misconfiguration issues can result in provisioning failures. Some of them are documented below.

The platform provides tooling for "pre-flight" checks on the nodes before initiating provisioning. These pre-flight checks are designed to quickly detect environmental or misconfiguration issues that will result in cluster provisioning issues.

Important

Please initiate provisioning ONLY after the pre-flight checks have passed successfully.


RCTL CLI

Users that are using the RCTL CLI with declarative cluster specifications for cluster provisioning and lifecycle management may encounter errors that will be presented by RCTL. Some of the commonly encountered issues are listed below

Credentials Upload

This occurs when the RCTL CLI is unable to secure copy the credentials and the conjurer binary to the remote node(s).

./rctl apply -f singlenode-test.yaml
{
  "taskset_id": "lk5owy2",
  "operations": [
    {
      "operation": "ClusterCreation",
      "resource_name": "singlenode-test1",
      "status": "PROVISION_TASK_STATUS_PENDING"
    },
    {
      "operation": "BlueprintSync",
      "resource_name": "singlenode-test1",
      "status": "PROVISION_TASK_STATUS_INPROGRESS"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

Downloading Installer And Credentials
Copying Installer and credentials to node:  35.84.184.226
scpFileToRemote() failed, err: failed to dial: dial tcp 35.84.184.226:22: connect: operation timed out%

Incorrect SSH Details

Provisioning will fail if incorrect SSH details are specified in the cluster specification file.

Error: Error performing apply on cluster "vyshak-mks-ui-test1": server error [return code: 400]: {"operations":null,"error":{"type":"Processing Error","status":400,"title":"Processing request failed","detail":{"Message":"Active Provisioning is in progress, cannot initiate another\n"}}}

Incorrect Node Details

Provisioning will fail if incorrect node details are specified in the cluster specification file.

Error: Error performing apply on cluster "vyshak-singlenode-test1": server error [return code: 400]: {"operations":null,"error":{"type":"Processing Error","status":400,"title":"Processing request failed","detail":{"Message":"Cluster with name \"vyshak-singlenode-test1\" is partially provisioned\n"}}}

Managed Storage

The customers using the rook-ceph storage node must deploy the default-upstream blueprint to the cluster.

Step 1: Verify Blueprint Sync

The rook-ceph storage is provided as an add-on with default-upstream blueprint, thus users can verify the rook-ceph managed storage deployment using the blueprint sync icon. Refer Update Blueprint to know more about update blueprint sync status

Step 2: Verify Health of Pods

On successful blueprint sync, users can view the rook-ceph pods running as shown in the below example:

kubectl -n rook-ceph get pod
11NAME                                             READY   STATUS      RESTARTS   AGE
12csi-cephfsplugin-4r8c5                           3/3     Running     0          4m1s
13csi-cephfsplugin-provisioner-b54db7d9b-mh7mb     6/6     Running     0          4m
14csi-rbdplugin-6684r                              3/3     Running     0          4m1s
15csi-rbdplugin-provisioner-5845579d68-sq7f2       6/6     Running     0          4m1s
16rook-ceph-mgr-a-f576c8dc4-76z96                  1/1     Running     0          3m8s
17rook-ceph-mon-a-6f6684764f-sljtr                 1/1     Running     0          3m29s
18rook-ceph-operator-75fbfb7756-56hq8              1/1     Running     1          17h
19rook-ceph-osd-0-5c466fd66f-8lsq2                 1/1     Running     0          2m55s
20rook-ceph-osd-prepare-oci-robbie-tb-mks1-b9t7s   0/1     Completed   0          3m5s
21rook-discover-5q2cq                              1/1     Running     1          17h

Common Issues

Provisioning Challenges with Conjurer Preflight Tests

Although "conjurer" provides a built-in battery of "preflight tests" that can be used to verify the environment and configuration, there are some scenarios where provisioning can fail.

Host Firewall

If your instances (for the nodes) have a host firewall such as firewall or iptables rules, it may be silently drop all packets destined for the Controller. This will result in provisioning failure. Ensure that the host firewall is configured to allow outbound communications on tcp/443 to the controller.

Inter Node Communication

Ensure that the host firewall is configured to allow inter node communication so that pod's can reach to the api server

MTU

The Maximum Transmission Unit (MTU) is the largest block of data that can be handled at Layer-3 (IP). MTU usually refers to the maximum size a packet can be. Certain MTU/MSS settings can result in fragmentation related issues with mTLS connections between the agents and the controller.

Unstable Network

Unstable or unreliable network connectivity. Remote cluster provisioning in remote, low bandwidth locations with unstable networks can be very challenging. Review how the retry and backoff mechanisms work by default and how they can be customized to suit your requirements.


Cluster Provision/Upgrade Failures

When a cluster provision/upgrade or node addition to an exiting cluster fails, perform the below steps:

  1. ssh to faculty node where provision/upgrade/addition failed
  2. Run systemctl status salt-minion-rafay to check the minion status (active/inactive)
  3. If the Salt minion is not active, restart it using the command systemctl restart salt-minion-rafay
  4. To check for any errors, verify the logs of the active minion from the file cat /opt/rafay/salt/var/log/salt/minion

Note

It will help the support team if you are able to zip the log files and share with them.


Connectivity Issues Between Pods and Kubernetes Apiserver in Multi-Node Setup

Failure Scenario

Provision Failed

In a restricted environment, a common challenge arises when firewalld hinders Pods from reaching the apiserver. This can lead to failures during cluster provisioning or upgrades. If you encounter such issues, follow these troubleshooting steps:

  1. Inspect the container logs of the affected Pods to identify any errors. Execute the command:
    kubectl logs <pod-name>
    
  2. Check the status of firewalld on your machine using the following commands:

    sudo systemctl status firewalld
    
    Ensure that firewalld is appropriately configured to allow inter-node communication in a multi-node setup, enabling Pods to reach the api server. If you encounter issues and suspect that firewalld might be causing the problem, you can temporarily stop and disable it using the following commands:
    sudo systemctl stop firewalld
    sudo systemctl disable firewalld
    
    After stopping and disabling firewalld, check if the connectivity issues persist. If the problem is resolved, you may need to adjust the firewalld configuration to allow the necessary communication between your pods on worker node to Kubernetes API server pod on the master .

  3. Investigate potential issues with iptables. As a troubleshooting step, attempt to resolve it by clearing out iptables using the command:

    iptables -F
    

  4. Check the minion logs for debugging purposes. The minion logs are located at /opt/rafay/salt/var/log/salt/minion. In case of connectivity failures, the logs might indicate the node's inability to connect to the master node. A sample failure scenario in the logs could resemble the following:
    error execution phase preflight: couldn't validate the identity of the API Server:
    Get "https://k8master.service.consul:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup k8master.service.consul on 127.0.0.53:53: no such host
    
    If you encounter such entries, consider investigating firewall settings, network configurations, or any other connectivity issues that might be hindering the communication between nodes in your Kubernetes cluster.

Connectivity Issues Between Node and Rafay Controller

Ensure whitelisting in place in your datacenter’s firewall to allow the hosts to make outbound TLS (port 443) connection to Rafay.

Whitelisting IP/FQDN to our controller can be found here.


Pods on different nodes are not able to communicate together

Pods on different nodes might experience communication failures for two reasons. First, a firewall could be blocking the communication. Second, there could be a discrepancy in the Maximum Transmission Unit (MTU) values between worker and master nodes. MTU is like the maximum size of a message that can be sent.

To address these communication issues:

  • Check for firewalls: Verify that no firewalls are preventing pods from communicating across nodes.

  • Verify MTU consistency: Compare the MTU values on worker and master nodes. Ensure that the MTU configuration on the network interfaces of both types of nodes is accurate and consistent. This will help ensure smooth communication between pods.


Resolving Intermittent Worker Node Down During MKS Cluster Upgrade

If you encounter intermittent failures during a Kubernetes (MKS) Cluster upgrade with a message indicating that the worker node is down, follow these steps:

1 Status of control channel for <Node_name>: Down
2 Status of control channel for <Node_name>: Up
3 Status of control channel for <Node_name>: Up
4 Status of control channel for <Node_name>: Up

Run the following commands on the MKS cluster node that is down:

$ systemctl restart salt-minion
$ systemctl restart chisel_service

This should resolve the issue and allow you to proceed with the upgrade.


Error During Node Addition in the Cluster When running Conjurer

If you encounter the following error while running Conjurer during the process of adding a node to the cluster:

!ERROR! Failed to detect Operating System. None

Ensure that the Operating System of the machine where Conjurer is executed is supported. Refer to the list of supported OS here


Cluster Provisioning Fails - Node Not Found

stderr: Error from server (NotFound): nodes \"$NODE_NAME\" not found

This error is triggered when a node hostname contains an invalid character. According to Kubernetes naming conventions, the node name must adhere to certain rules .

To resolve this issue, recreate the node with a valid hostname and then restart the provisioning process. Ensure the new hostname complies with Kubernetes naming guidelines


Worker Node in NotReady State

Scenario 1: Kubelet Service NOT running

If a worker node is observed to be in a NotReady state, it could be attributed to the kubelet service not running on the node. This scenario often arises when swap is enabled on the worker node.

To address this issue:

  • Execute the following command to check the status of the kubelet service on the worker node:
sudo systemctl status kubelet

If the status is NotReady, determine if swap is enabled on the worker node. If swap is active, it can hinder the proper functioning of the kubelet service.

  • Disable swap and restart Kubelet service

It is recommended to ensure that swap is disabled on nodes within a Kubernetes cluster for optimal performance and reliability.

Scenario 2: Worker Node Communication with Master Nodes

Ensure that all required ports are accessible between both master and worker nodes.

To check whether the necessary ports between nodes are open, use the below command

nc -zv  <IP>:<PORT>

If the worker node can reach the master node and the necessary ports are open, the command will execute successfully.

Refer Inter Node Communication for more information


Support

If you are unable to resolve the issue yourself, please contact Support or via the provided private Slack channel for your organization. The support organization is available 24x7 and will be able to assist you immediately.

Please make sure that you have downloaded the "error log file" that was shown during failure. Provide this to the support team for troubleshooting.


Remote Diagnosis and Resolution

For customers using the SaaS Controller, with your permission, as long as the nodes are operational (i.e. running and reachable), the support team can remotely debug, diagnose and resolve issues for you. Support will inform the customer if the underlying issue is due to misconfiguration (e.g. network connectivity) or environmental issues (e.g. bad storage etc).

Important

Support DOES NOT require any form of inbound connectivity to perform remote diagnosis and fixes.