CrashLoopBackOff (Mongo in Docker/Kubernetes) - Failed to start up WiredTiger under any compatibility version

问题

I'm suddenly facing some issues in my Kubernetes application (with no event to explain it). The application has been working properly during one year but now I'm getting a CrashLoopBackOff status.

---

IMPORTANT UPDATE:

I cannot update the Mongo replication controller in GKE, because when I commit the changes in mongo.yml (from GIT) all workloads update except mongo-controller (which is down).

In GKE in Workloads/Mongo-controller/Managed pods I can see that the "Created on" date is some days ago when the app was up. The rest of pods are updating with my commits. I don't want to delete the Mongo pod, because I suppose that we'd lost the database info/content. (The developer who created the cluster pipeline didn't schedule a backup).

---

Database: MongoDB (latest, not sure what was the one running properly) OS: Pod running on Ubuntu 18.04 CLuster: Google Cloud Kubernetes Engines (a.k.a GKE)

Kubectl get pods

mongo-controller-dgkkg 0/1 CrashLoopBackOff 1199 4d6h

Logs of Mongo POD

Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'

No TransportLayer configured during NetworkInterface startup"}

Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.

MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"mongo-controller-dgkkg"}

Build Info","attr":{"buildInfo":{"version":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}

Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}

Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}

Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}

Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]

Opening WiredTiger","attr":{"config":"create,cache_size=1336M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress]

Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade.

Terminating.","attr":{"reason":"95: Operation not supported"}}

Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1101}}

\n\n***aborting after fassert() failure\n\n

My Mongo.yml:

apiVersion: v1
kind: Service
metadata:
  name: mongo
  namespace: $KUBE_NAMESPACE-$CI_ENVIRONMENT_SLUG
  labels:
    name: mongo
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    name: mongo
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: mongo-controller
  namespace: $KUBE_NAMESPACE-$CI_ENVIRONMENT_SLUG
  labels:
    name: mongo
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: mongo
    spec:
      containers:
      - image: mongo
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
      volumes:
        - name: mongo-persistent-storage
          gcePersistentDisk:
            pdName: mongo-disk-$CI_ENVIRONMENT_SLUG
            fsType: ext4

PD: Maybe I should update my ReplicationController to Deployment (recommended), but being a database container we always configure it in that way. However I tried that, and nothing changed.

回答1:

I solved this issue editing the Replication Controller online from the Google Cloud Console.

Access to: "Kubernetes Engine" > "Workload" > "mongo-controller" > "Managed pods" > "mongo-controller-XXXXX"

...and press EDIT button (in the top navbar). You can edit the configuration online in real time. I simply specified the Mongo version (4.2.10) in the image, and everything woked as expected.

    spec:
      replicas: 1
      selector:
        name: mongo
      template:
        metadata:
          creationTimestamp: null
          labels:
            name: mongo
        spec:
          containers:
          - image: mongo: 4.2.10

          (...)

来源：https://stackoverflow.com/questions/64458588/crashloopbackoff-mongo-in-docker-kubernetes-failed-to-start-up-wiredtiger-un