My kubernetes pods keep crashing with “CrashLoopBackOff” but I can't find any log

Question

This is what I keep getting:

[root@centos-master ~]# kubectl get pods
NAME               READY     STATUS             RESTARTS   AGE
nfs-server-h6nw8   1/1           


        

Answer 1

As @Sukumar commented, you need to have your Dockerfile have a Command to run or have your ReplicationController specify a command.

The pod is crashing because it starts up then immediately exits, thus Kubernetes restarts and the cycle continues.

Answer 2

I solved this problem I increased memory resource

  resources:
          limits:
            cpu: 1
            memory: 1Gi
          requests:
            cpu: 100m
        memory: 250Mi 

Answer 3

Try rerunning the pod and running

 kubectl get pods --watch

to watch the status of the pod as it progresses.

In my case, I would only see the end result, 'CrashLoopBackOff,' but the docker container ran fine locally. So I watched the pods using the above command, and I saw the container briefly progress into an OOMKilled state, which meant to me that it required more memory.

Answer 4

I had similar issue but got solved when I corrected my zookeeper.yaml file which had the service name mismatch with file deployment's container names. It got resolved by making them same.

apiVersion: v1
kind: Service
metadata:
  name: zk1
  namespace: nbd-mlbpoc-lab
  labels:
    app: zk-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zk-1
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zk-deployment
  namespace: nbd-mlbpoc-lab
spec:
  template:
    metadata:
      labels:
        app: zk-1
    spec:
      containers:
      - name: zk1
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zk1

Answer 5

kubectl -n <namespace-name> describe pod <pod name>

kubectl -n <namespace-name> logs -p  <pod name> 

Answer 6

If you have an application that takes slower to bootstrap, it could be related to the initial values of the readiness/liveness probes. I solved my problem by increasing the value of initialDelaySeconds to 120s as my SpringBoot application deals with a lot of initialization. The documentation does not mention the default 0 (https://kubernetes.io/docs/api-reference/v1.9/#probe-v1-core)

service:
  livenessProbe:
    httpGet:
      path: /health/local
      scheme: HTTP
      port: 8888
    initialDelaySeconds: 120
    periodSeconds: 5
    timeoutSeconds: 5
    failureThreshold: 10
  readinessProbe:
    httpGet:
      path: /admin/health
      scheme: HTTP
      port: 8642
    initialDelaySeconds: 150
    periodSeconds: 5
    timeoutSeconds: 5
    failureThreshold: 10

A very good explanation about those values is given by What is the default value of initialDelaySeconds.

The health or readiness check algorithm works like:

1. wait for initialDelaySeconds
2. perform check and wait timeoutSeconds for a timeout if the number of continued successes is greater than successThreshold return success
3. if the number of continued failures is greater than failureThreshold return failure otherwise wait periodSeconds and start a new check

In my case, my application can now bootstrap in a very clear way, so that I know I will not get periodic crashloopbackoff because sometimes it would be on the limit of those rates.