Relation between preStop hook and terminationGracePeriodSeconds

Question

Basically I am trying to do is play around with pod lifecycle and check if we can do some cleanup/backup such as copy logs before the pod terminates.

What I need : Copy logs/heapdumps from container to a hostPath/S3 before terminating

What I tried:

I used a preStop hook with a bash command to echo a message (just to see if it works !!). Used terminationGracePeriodSeconds with a delay to preStop and toggle them to see if the process works. Ex. keep terminationGracePeriodSeconds:30 sec (default) and set preStop command to sleep by 50 sec and the message should not be generated since the container will be terminated by then. This works as expected.

My questions:

• what kind of processes are allowed(recommended) for a preStop hook? As copying logs/heapdumps of 15 gigs or more will take a lot of time. This time will then be used to define terminationGracePeriodSeconds
• what happens when preStop takes more time than the set gracePeriod ? (in case logs are huge say 10 gigs)
• what happens if I do not have any hooks but still set terminationGracePeriodSeconds ? will the container remain up until that grace time ?

I found this article which closely relates to this but could not follow through https://github.com/kubernetes/kubernetes/issues/24695

All inputs appreciated !!

Answer 1

what kind of processes are allowed(recommended) for a preStop hook? As copying logs/heapdumps of 15 gigs or more will take a lot of time. This time will then be used to define terminationGracePeriodSeconds

Anything goes here, it's more of an opinion and how you would like your pods to linger around. Another option is to let your pods terminate and store your data in some place (i.e, AWS S3, EBS) where data will persist past the pod lifecycle then use something like Job to clean up the data, etc.

what happens when preStop takes more time than the set gracePeriod? (in case logs are huge say 10 gigs)

Your preStop will not complete which may mean incomplete data or data corruption.

what happens if I do not have any hooks but still set terminationGracePeriodSeconds ? will the container remain up until that grace time ?

This explains the sequence (from here):

• A SIGTERM signal is sent to the main process in each container, and a “grace period” countdown starts.
• If a pod has a preStop hook its invoked inside the container.
• If a container doesn’t terminate within the grace period, a SIGKILL signal will be sent and the container.