View log files of crashed pods in kubernetes

Question

Any idea to view the log files of a crashed pod in kubernetes? My pod is listing it\'s state as \"CrashLoopBackOff\" after started the replicationController. I search the availa

Answer 1

kubectl logs command only works if the pod is up and running. If they are not, you can use the kubectl events command.

kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'

By default it does not sort the events, hence the --sort-by flag.

Answer 2

In many cases, kubectl logs <podname> --previous is returning:

Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found

So you can try to check in the namespace's events (kubectl get events ..) like @alltej showed.

If you don't find the reason for the error with kubectl logs / get events and you can't view it with external logging tool I would suggest:

1 ) Check on which node that pod was running on with:

$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName

NAME              STATUS    NODE
failed-pod-name   Pending   dns-of-node

(If you remove the <pod-name> you can see other pods in the namespace).

2 ) SSH into that node and:

A ) Search for the failed pod container name in /var/log/containers/ and dump its .log file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.

B ) If previous step doesn't help try searching for latest System level errors by running:
sudo journalctl -u kubelet -n 100 --no-pager.

Answer 3

There was a bug in kubernetes that prevents logs obtaining for pods in CrashLoopBackOff state. Looks like it was fixed. Here issue on github with additional information

Answer 4

Assuming that your pod still exists:

kubectl logs <podname> --previous

$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.