问题
All of a sudden, I cannot deploy some images which could be deployed before. I got the following pod status:
[root@webdev2 origin]# oc get pods
NAME READY STATUS RESTARTS AGE
arix-3-yjq9w 0/1 ImagePullBackOff 0 10m
docker-registry-2-vqstm 1/1 Running 0 2d
router-1-kvjxq 1/1 Running 0 2d
The application just won't start. The pod is not trying to run the container. From the Event page, I have got Back-off pulling image "172.30.84.25:5000/default/arix@sha256:d326. I have verified that I can pull the image with the tag with docker pull.
I have also checked the log of the last container. It was closed for some reason. I think the pod should at least try to restart it.
I have run out of ideas to debug the issues. What can I check more?
回答1:
You can use the 'describe pod' syntax
For OpenShift use:
oc describe pod <pod-id>
For vanilla Kubernetes:
kubectl describe pod <pod-id>
Examine the events of the output. In my case it shows Back-off pulling image coredns/coredns:latest
In this case the image coredns/coredns:latest can not be pulled from the Internet.
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5m 5m 1 {default-scheduler } Normal Scheduled Successfully assigned coredns-4224169331-9nhxj to 192.168.122.190
5m 1m 4 {kubelet 192.168.122.190} spec.containers{coredns} Normal Pulling pulling image "coredns/coredns:latest"
4m 26s 4 {kubelet 192.168.122.190} spec.containers{coredns} Warning Failed Failed to pull image "coredns/coredns:latest": Network timed out while trying to connect to https://index.docker.io/v1/repositories/coredns/coredns/images. You may want to check your internet connection or if you are behind a proxy.
4m 26s 4 {kubelet 192.168.122.190} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "coredns" with ErrImagePull: "Network timed out while trying to connect to https://index.docker.io/v1/repositories/coredns/coredns/images. You may want to check your Internet connection or if you are behind a proxy."
4m 2s 7 {kubelet 192.168.122.190} spec.containers{coredns} Normal BackOff Back-off pulling image "coredns/coredns:latest"
4m 2s 7 {kubelet 192.168.122.190} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "coredns" with ImagePullBackOff: "Back-off pulling image \"coredns/coredns:latest\""
Additional debuging steps
- try to pull the docker image and tag manually on your computer
- Identify the node by doing a 'kubectl/oc get pods -o wide'
- ssh into the node (if you can) that can not pull the docker image
- check that the node can resolve the DNS of the docker registry by performing a ping.
- try to pull the docker image manually on the node
- If you are using a private registry, check that your secret exists and the secret is correct. Your secret should also be in the same namespace. Thanks swenzel 6 Some registries have firewalls that limit ip address access. The firewall may block the pull
- Some CIs create deployments with temporary docker secrets. So the secret expires after a few days (You are asking for production failures...)
回答2:
Have you tried to edit to see what's wrong (I had the wrong image location)
kubectl edit pods arix-3-yjq9w
or even delete your pod?
kubectl delete arix-3-yjq9w
回答3:
I forgot to push the image tagged 1.0.8 to the ECR (AWS images hub)... If you are using Helm and upgrade by:
helm upgrade minta-user ./src/services/user/helm-chart
make sure that image tag inside values.yaml is pushed (to ECR or Docker Hub, etc) for example: (this is my helm-chart/values.yaml)
replicaCount: 1
image:
repository:dkr.ecr.us-east-1.amazonaws.com/minta-user
tag: 1.0.8
you need to make sure that the image:1.0.8 is pushed!
回答4:
Run docker login
Push the image to docker hub
Re-create pod
This solved the problem for me. Hope it helps.
来源：https://stackoverflow.com/questions/34848422/how-to-debug-imagepullbackoff