How does pod replica scaling down work in Kubernetes Horizontal Pod Autoscaler?

Question

My understanding is that in Kubernetes, when using the Horizontal Pod Autoscaler, if the targetCPUUtilizationPercentage field is set to 50%, and the average CPU uti

Answer 1

So you have two questions in there and let me address one by one. The first part - if a pod in a replica set is consuming let's say 10% then will Kubernetes kill that pod? The answer is Yes. Kubernetes is not looking at the individual pods but at an average of that metric across all pods in that replica set. Also the scaling down is gradual as explained here

The second part of the question - how does your application behave gracefully when a pod is about to be killed and it is still serving some requests? This can be handled by the grace period of the pod termination and even better if you implement a PreStop hook - which will allow you to do something like stop taking incoming requests but process existing requests. The implementation of this will vary based on the language runtime you are using, so I won't go in the details here.

Lastly - one scenario you should consider is what if VM on which pod was running goes down abruptly - you have no chance to execute PreStop hook! I think the application needs to be robust enough to handle failures.