Nodes are a vital component of a Kubernetes cluster and are responsible for running the pods. Depending on your cluster setup, a node can be a physical or a virtual machine. A cluster typically has one or multiple nodes, which are managed by the control plane.
Because nodes do the heavy lifting of managing the workload, you want to make sure all your nodes are running correctly. The
kubectl get nodes command can be used to check the state of your nodes.
A node with a
NotReady status means it can’t be used to run a pod because of an underlying issue. It’s essentially used to debug a node in the
NotReady state so that it doesn’t lie unused.
In this article, you’ll learn a few possible reasons why a node might enter the
NotReady state and how you can debug it.
The notready state
As mentioned earlier, each node in a cluster is used to run pods. Before a pod is scheduled on a node, Kubernetes checks whether the node is capable of running the pod or not. The
STATUS column in the output of
kubectl get nodes represents the status. The possible values in this column are:
Ready: The node is healthy and ready to accept pods.
NotReady: The node has encountered some issue and a pod cannot be scheduled on it.
SchedulingDisabled: The node is marked as unschedulable. This can be done using the kubectl cordon command.
Unknown: The node is unreachable by the control plane.
Having a node in the
NotReady state means that the node is effectively unused and will accumulate costs without participating in running pods. Furthermore, losing a node can negatively impact your production workload.
In order for your application to run smoothly, you must debug them quickly.
Possible causes of the notready state
There can be various reasons why a node might enter the
NotReady state. This section will review some of the most common reasons for this error.
Scarcity of resources
To operate normally, a node must have sufficient disk space, memory, and sufficient processing ability. If a node is running low on disk space or the available memory is low, it will go into the
NotReady state. If pressure exists on the processes, eg too many processes are running on the node, it will also change to the
If the network has not been correctly configured on the node or it can’t reach the internet, the node will be unable to communicate with the master node and will be listed as
Issue with kubelet process
kubelet is an agent that runs on each node. It is responsible for communicating with the Kubernetes API server and registering the nodes. If kubelet crashes or stops on the node, it will not be able to communicate with the API Server and will be in the
Issue with kube-proxy
kube-proxy is a network proxy that runs on each node and maintains the network rules. These rules allow network communication to your pods from inside or outside your cluster. If kube-proxy crashes or stops, the node will be in the
Vendor specific issues
Suppose you’re using a cloud-hosted solution like GKE or EKS. In that case, some vendor-specific issues may be preventing your nodes from operating normally and communicating with the control plane. These issues could be IAM misconfiguration, misconfigured network rules, etc.
Debugging the notready state
As you can see, the
NotReady status can be caused by a multitude of issues. This section will help you identify the root cause of the problem. However, it’s essential to understand that how you go about fixing these issues depends on the exact cause and your cluster setup. There are no one-size-fits-all solutions. But, once you identify the root cause, it should be easier to resolve it.
Check the kube-proxy pod
First, ensure that each node has exactly one
kube-proxy pod and is in the
The output might look like this:
|NAME||READY||STATUS||AGE||IP||NODE||NOMINATED NODE||READINESS GATES|
|kube-proxy-nhbtp||1/1||Running||2 (11h ago)||2d16h||192.168.99.10||1 my-cluster||<none>||<none>|
|kube-proxy-tkmsk||1/1||Running||2 (11h ago)||2d16h||192.168.99.10||3 my-cluster-m03||<none>||<none>|
|kube-proxy-vk4ch||1/1||Running||2 (11h ago)||2d16h||192.168.99.10||2 my-cluster-m02||<none>||<none>|
If any one pod is in some state other than
Running, use the following command to get more information:
The Events section logs the various events on the pod, and it could be an excellent place to start looking for any mishaps.
You can get access to the pod logs by running the following command:
The logs and the events list is a good place to start looking for any issues.
If your node does not have a
kube-proxy pod, then you need to inspect the
kube-proxy daemonset, which is responsible for running one kube-proxy pod on each node.
The output of this command might reveal any possible issue with the daemonset.
Verify resources are available
Run the following command to get detailed information about a node that is not ready:
In the output, the
Conditions section shows if the node is running out of resources or not.
The following conditions are available:
True, it indicates that the node is running out of memory.
Truevalue in this field indicates that the node lacks enough space.
PIDPressure: If too many processes are running on the node, this field will be
NetworkUnavailable: If the network for the node is not correctly configured, this will be
Ready: If the node is healthy and ready to accept pods, this will be
True. In this field, a
Falseis equivalent to the
NotReadystatus in the
get nodesoutput. It can also have the
Unknownvalue, which means the node controller has not heard from the node in the last
node-monitor-grace-period(defaults to 40 seconds).
If any one of the first four conditions is
True, you have identified the problem.
Verify kubelet is running
If all the
Conditions fields show
Unknown, it might hint that the kubelet process on the node has run into some issues.
To debug this, first SSH into the node and check the status of the kubelet process. If it’s running as a systemd service, use the following command:
Active field shows
inactive (dead), it means the kubelet process has stopped.
To reveal the possible reason for the crash, check the logs with the following command:
Once the issue is fixed, restart kubelet with:
Verify network communication with the control plane
Conditions field shows
NetworkUnavailable, it indicates an issue in the network communication between the node and the control plane.
A few possible fixes:
- If the node is configured to use a proxy, verify that the proxy allows access to the API server endpoints.
- Ensure that the route tables are appropriately configured to avoid blocking communication with the API server.
- If you’re using a cloud provider like AWS, verify that no VPC network rules block communication between the control plane and the node.
You can run the following command from within the node to verify that it can reach the API server.
If the output shows
succeeded, then network communication is working correctly.
Vendor specific debugging
If you’re using a cloud provider like EKS, or GKE, sometimes it’s worth looking into vendor-specific issues if you’ve exhausted all other debugging techniques. EKS has an extremely detailed guide that you can follow.
GKE provides an auto repair feature that can attempt to repair a node that has been in the
NotReady state for a given amount of time. If all else fails, you can always get in touch with your cloud provider for more assistance.
Having a node in the
NotReady state is undesirable and needs to be fixed immediately. However, there are multiple reasons this might occur, and it can be challenging to pinpoint the exact cause. This article discussed some common reasons you may encounter the
NotReady command and solutions for it.
The earlier you can catch nodes entering the
NotReady state, the higher your chances of quickly debugging it. You can also see all the events leading up to this, allowing you to quickly identify and solve the issue.
If you're looking to streamline your debugging processes or build dashboards to help monitor your Kubernetes clusters, check out Airplane. Airplane allows users to transform scripts, queries, APIs, and more into powerful internal UIs and workflows. Airplane's engineering workflows solution is geared toward engineering use cases and can help you get started quickly and easily when building internal workflows designed to help debug your Kubernetes nodes.