Solution To Apache Triggering OOM-Killer
Update: We switched to Prefork apache and have been experiencing significantly fewer problems.In this instance, we were getting hit with web crawlers according to the access log and the syslog was saying this: /usr/sbin/apach invoked oom-killer
This issue occurs when apache opens up too many child processes, uses up too much memory, then OOM killer starts shutting down random processes like mysql.
The problem ended up being that our apache2.conf did not have any configuration limiting the number of processes that Apache could open. It might have had a default, but the default was too high. We were using Apache ITK MPM and the apache2.conf only mentioned prefork, worker, and event MPMs. You can check which MPM you are running with "apache2 -V".
Adding this to the configuration solved the problem:
<IfModule mpm_itk_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 75
MaxClients 75
MaxRequestsPerChild 5000
</IfModule>
At first I had the ServerLimit and MaxClients set to 150 and it seemed like apache was ignoring ServerLimit and Maxclients. However, ITK MPM creates an extra fork for each request (according to their site), that's why there was 200+ processes running for my limit of 150. So, the process count is going to be 2x the limit you set.