Apache shutdown - Couldn't grab mutex

Question

For some reasons unknow to me, Apache is crashing.

This is the apache log at the moment of the crash:

[Sat Jun 02 02:38:05.196006 2018] [mpm_prefork:emerg] [pid 1122] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:05.196006 2018] [mpm_prefork:emerg] [pid 1116] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:05.198767 2018] [mpm_prefork:emerg] [pid 1115] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:05.199523 2018] [mpm_prefork:emerg] [pid 3924] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:05.204189 2018] [mpm_prefork:emerg] [pid 1937] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:05.207362 2018] [mpm_prefork:emerg] [pid 1120] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Sat Jun 02 02:38:06.057310 2018] [core:alert] [pid 1099] AH00050: Child 1115 returned a Fatal error... Apache is exiting!
[Sat Jun 02 02:38:06.057483 2018] [:emerg] [pid 1099] AH02818: MPM run failed, exiting

The last thing I did on the server lately was to install certbot to general HTTPS certificates for my sites.

Answer 1

I was experiencing the same error on Ubuntu 17.10 with Apache 2.4.27.

It appears to have to do with Apache talking to the OS about file locking. When I went into /etc/apache2/apache2.conf and added Mutex flock to the Mutex section (near the ServerRoot section), it seems to have solved the problem.

Unfortunately, all I can say is that this uses the flock() function (a system call?) to handle file locking. Per https://httpd.apache.org/docs/2.4/mod/core.html#mutex

So it's a bit of a black magic fix in that I personally am not familiar enough with Apache's file locking behavior to say why it works, and I can only say it works because the Apache crash seems to have stopped, but only thus far. Nevertheless, it may help you and seems fairly low-risk.

Answer 2

You haven't shared the complete log trace. It could be:

[Sun Jul 21 15:49:43 2019] [error] (12)Cannot allocate memory: fork: Unable to fork new process

This is due to low memory and apache failed to create new processes. It also depends on the number of processes you have configured it to fork. It happens when there is too much traffic on to the website.

Answer 3

While Sean solution is valid, I learned another way to solve this problem.

First, you should know that this problem is related to using Certbot and SSL certificates on Ubuntu.

Now, to solve the problem without doing anything you're unsure of, simply do this (as root or with sudo):

1. Create the following file: /etc/apache2/conf-available/mutex-file.conf
2. Add Mutex file:${APACHE_LOCK_DIR} default inside the newly created file
3. Enable new configuration file with a2enconf mutex-file.conf
4. Restart Apache with systemctl restart apache2

But hey, it's already enabled by default! Why make a file just for that?

I though the same when I found this solution.

By one wonder or another, it seems that once your start using SSL certificates, this configuration line is somehow "ignored".
Thus, Apache fail to find the lock dir for the mutex file.

Just adding (remove comment) this line into /etc/apache2/apache2.conf isn't working either.
You have to specifically create a configuration file and enable it.
Else it won't solve the problem.