Kubernetes - Rolling update killing off old pod without bringing up new one

Question

I am currently using Deployments to manage my pods in my K8S cluster.

Some of my deployments require 2 pods/replicas, some require 3 pods/replicas and some of them requi

Answer 1

As answered already, you can set the maxUnavailable to 0 to achieve the desired result. A couple of extra notes:

1. You should not expect this to work when using a stateful service that mounts a single specific volume that is to be used by the new pod. The volume will be attached to the soon-to-be-replaced pod, so won't be able to attach to the new pod.

2. The documentation notes that you cannot set this to 0 if you have set .spec.strategy.rollingUpdate.maxSurge to 0.

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-unavailable

Answer 2

It appears to be the maxUnavailable: 1; I was able to trivially reproduce your experience setting that value, and trivially achieve the correct experience by setting it to maxUnavailable: 0

Here's my "pseudo-proof" of how the scheduler arrived at the behavior you are experiencing:

Because replicas: 1, the desired state for k8s is exactly one Pod in Ready. During a Rolling Update operation, which is the strategy you requested, it will create a new Pod, bringing the total to 2. But you granted k8s permission to leave one Pod in an unavailable state, and you instructed it to keep the desired number of Pods at 1. Thus, it fulfilled all of those constraints: 1 Pod, the desired count, in an unavailable state, permitted by the R-U strategy.

By setting the maxUnavailable to zero, you correctly direct k8s to never let any Pod be unavailable, even if that means surging Pods above the replica count for a short time

Answer 3

with Strategy Type set to RollingUpdate a new pod is created before the old one is deleted even with a single replica. Strategy Type Recreate kills old pods before creating new ones https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment