Spring Boot app + Kubernetes liveness/readiness checks

Question

I\'m building a few Spring Boot microservices that are getting deployed in a Kubernetes (AKS specifically) cluster. I was planning on setting the probePaths for the

Answer 1

ReadinessProbe - is the app ready to handle requests?

Use a health check to check if the app is ready to handle new requests. This can be implemented in /actuator/health. Also see StartupProbe below.

Under high load?

If your app is under high load, it may not be able to respond on health check in time, resulting in ReadinessProbe to fail. Consider using Horizontal Pod Autoscaler to get more replicas to handle the load.

LivenessProbe - is the app deadlocked?

If your app is in a unrecoverable state, it is best if it can terminate itself, e.g. using java.lang.System.exit(1). If the app can be deadlocked, unable to proceed, consider implementing an endpoint for LivenessProbe, this may be the same as for the ReadinessProbe.

Not responding to readiness in a long time

If your app haven't responded to the ReadinessProbe in a long time, e.g. many minutes, something is probably wrong (unless you expect this to happen for your app), then you should probably also have /actuator/health as your LivenessProbe but with a higher failureThreshold and a high initialDelaySeconds (e.g. a few minutes)

StartupProbe - better alternative on Kubernetes 1.16+

The ReadinessProbe is most useful during app startup, since it may need to load e.g. data before it is ready to receive requests - but ReadinessProbe is executed periodic during the pod lifecycle. StartupProbe is now a better alternative for slow starting apps in combination with LivenessProbe that is only active after StartupProbe. You may still need a ReadinessProbe to notify that the pod is ready to handle requests.

Depending on other services

If your app depends on other services, that are not healthy - it is better if your app can recover from those situations, when the backing service is up again, e.g. reconnect. Otherwise this will be a domino chain reaction if you have a chain of services that does not respond on ReadinessProbe or LivenessProbe because the last app in the chain have a problem. Consider to provide degraded service, notify that you are not in full service, maybe some of your endpoints still works correct.

Use Management Server Port

It is the kubelet on the same node that send probe requests. Consider to use a Management Server Port for probes. You don't need to expose this port to the Service, better to use one port for http and another for management.

Cloud provider Load Balancer Service health check

If you are using a Cloud Provider Load Balancer, it may do health checks on your services, and you may need to configure the path it sends health checks on, e.g. Google Cloud Platform defaults to /. This is a health check for the Service not for the individual Pod.