What killed my process and why?

Question

My application runs as a background process on Linux. It is currently started at the command line in a Terminal window.

Recently a user was executing the application

Answer 1

In my case this was happening with a Laravel queue worker. The system logs did not mention any killing so I looked further and it turned out that the worker was basically killing itself because of a job that exceeded the memory limit (which is set to 128M by default).

Running the queue worker with --timeout=600 and --memory=1024 fixed the problem for me.

Answer 2

We have had recurring problems under Linux at a customer site (Red Hat, I think), with OOMKiller (out-of-memory killer) killing both our principle application (i.e. the reason the server exists) and it's data base processes.

In each case OOMKiller simply decided that the processes were using to much resources... the machine wasn't even about to fail for lack of resources. Neither the application nor it's database has problems with memory leaks (or any other resource leak).

I am not a Linux expert, but I rather gathered it's algorithm for deciding when to kill something and what to kill is complex. Also, I was told (I can't speak as to the accuracy of this) that OOMKiller is baked into the Kernel and you can't simply not run it.