1. Contain Oom-killer

Contain Oom-killer

homer says doh

Amazon micro instances are a great starting option for small websites. Learn how to contain oom-killer that may make wrong decision.

Oom-killer as MySQL vs Apache Arbiter

When it comes to competition for memory oom-killer steps in. I run Apache and MySQL on the same box. You probably shouldn’t do this, but I thought that on my tiny setup they will get along. But I was wrong. When first users came in, the system quickly ran out of memory:

Many httpd processes depleted all memory:

And guess what? oom-killer killed the mysqld process:

Of course, mysqld_safe noticed there was no mysqld process and tried to restart it:

But after a few tries it failed, there was no memory to allocate for the buffer pool:

Terrible enough, isn’t it?

Let’s see how to prevent outages like this.

Limiting Memory Usage by Apache

The outage started because there were too many httpd processes. They basically consumed all memory and left nothing for MySQL. To solve the problem, Linux kernel calls oom-killer that tries to free some memory. MySQL is a known RAM consumer, so for oom-killer it’s the first in line. On the web server, I use the prefork Multi-Processing Module. It’s default on Amazon Linux and I don’t expect high traffic. The maximum number of httpd processes is controlled by MaxRequestWorkers. Its default is 256 which is too high for me. So I set it to 10 in /etc/httpd/conf/httpd.conf.

To prevent possible memory leaks, I set MaxConnectionsPerChild to 1000. After serving 1000 connections the child will die, so if any httpd module leaks the memory, after 1000 connections the memory will be freed. Default is 0, which means the child will never die i.e. it will accumulate all memory leaks.

Prevent Killing MySQL

Now, MySQL. Every process has a priority that oom-killer considers when it prepares to kill a process. You can see the priority in /proc file system:

By default, all user processes get 0. Possible values vary from -16 to 16. -16 means the oom-killer will kill the process the last, 16 means the process ist most likely to be killed first.

If /proc/$(pidof mysqld)/oom_adj is -17, oom-killer will never kill the process.

So I put a script in cron that periodically checks oom_adj of mysqld process and sets it to -17 if it’s different:

This is of course a temporary measure. See, my point about procrastination works. The next step will be to move MySQL to dedicated server and setup monitoring software.

Have a question? Ask the experts!

Previous Post Next Post