How to debug when Kubernetes nodes are in 'Not Ready' state

Question

I initialized the master node and add 2 worker nodes, but only master and one of the worker node show up when I run the following command:

kubectl get nodes
         


        

Answer 1

I was having similar issue because of a different reason:

Error:

cord@node1:~$ kubectl get nodes
NAME    STATUS     ROLES    AGE     VERSION
node1   Ready      master   17h     v1.13.5
node2   Ready         17h     v1.13.5
node3   NotReady      9m48s   v1.13.5

cord@node1:~$ kubectl describe node node3
Name:               node3
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  Ready            False   Thu, 18 Apr 2019 01:15:46 -0400   Thu, 18 Apr 2019 01:03:48 -0400   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.2.6
  Hostname:    node3

cord@node3:~$ journalctl -u kubelet

Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649047   54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649086   54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:50 node3 kubelet[54132]: E0418 01:24:50.649402   54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650816   54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650845   54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:55 node3 kubelet[54132]: E0418 01:24:55.651056   54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:57 node3 kubelet[54132]: I0418 01:24:57.248519   54132 setters.go:72] Using node IP: "192.168.2.6"

Issue:

My file: 10-calico.conflist was incorrect. Verified it from a different node and from sample file in the same directory "calico.conflist.template".

Resolution:

Changing the file, "10-calico.conflist" and restarting the service using "systemctl restart kubelet", resolved my issue:

NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    master   18h   v1.13.5
node2   Ready       18h   v1.13.5
node3   Ready       48m   v1.13.5