pod will not start due to “No nodes are available that match all of the following predicates:: Insufficient cpu”

Question

I\'m not sure why I\'m getting an error No nodes are available that match all of the following predicates:: Insufficient cpu (1).

I don\'t recall setting a

Answer 1

The error is pretty self explanatory, you are requesting 300m (100m per container in your pod) of cpu and your node doesn't have the budget to schedule it. (You seem to have just a 1 node cluster?)

You can describe the node to see how much cpu time is already scheduled.

I'm not sure what's adding those requests to your deployment though since you aren't stating those requirements in your template. Probably a resource quota.

Answer 2

I had got the same error but I created a cluster with just one node. So, I updated my cluster with min and max number of nodes with the following command:

gcloud container clusters update <mycluster-name> --enable-autoscaling --min-nodes=1 --max-nodes=15

And the very next moment, pod/pods changed status from Pending to Running.

Answer 3

There are additional containers running besides the ones specified in the config. These are provisioned by default with Kubernetes and run in the kube-system namespace which are not shown in the default namespace.

You can view all the pods by kubectl get pods --all-namespaces.

These additional containers are taking up 72% of the CPU quota of the single node...

Hence the 3 containers at 10% cpu quota would exceed 100% of the CPU quota (because 72% + (3 * 10) > 100%)...

As to why 72% of the containers is being allocated to the other containers - the question is asked here: Why does a single node cluster only have a small percentage of the cpu quota available?

Additional resources that may be usefull: How to reduce CPU limits of kubernetes system resources?

---

However, I was able to get the containers to run with sufficient CPU by adding additional nodes to the cluster. In addition, the high cpu instances seem to be more efficiently allocated on Google Cloud.