[clug] Out of Memory: Kill process 2689 (mysqld) score 33827 and children.

[Daniel Pittman]

Carlo Hamalainen <carlo.hamalainen at gmail.com> writes:

> I have a 512Mb Linode server running Ubuntu 8.10 which has been slowly
> apt-get upgraded towards 9.04. The server runs a few Wordpress blogs
> and a satchmo/django shop. Every week or so cpu usage goes to 100%,
> everything becomes unresponsive, and the console shows this:
>
> Out of Memory: Kill process 2961 (apache2) score 44591 andchildren.
> Out of memory: Killed process 3019 (apache2).
> Out of Memory: Kill process 2689 (mysqld) score 33827 and children.
> Out of memory: Killed process 2689 (mysqld).
> Out of Memory: Kill process 2699 (mysqld) score 33827 and children.
> Out of memory: Killed process 2699 (mysqld).
> Out of Memory: Kill process 2703 (mysqld) score 33827 and children.
> Out of memory: Killed process 2703 (mysqld).
> Out of Memory: Kill process 2961 (apache2) score 21690 and children.
> Out of memory: Killed process 3022 (apache2).
> Out of Memory: Kill process 15914 (mysqld) score 34048 and children.
> Out of memory: Killed process 15914 (mysqld).
>
> I have to hard reset the server as ssh is completely unresponsive.
>
> So, is this really a memory leak in mysqld?

Maybe.  It could also be that it happens to be one of the biggest processes,
so gets picked on first by the OOM killer.

> Other people [1] had a similar issue but the fix suggested was to change
> apache2's maxclients from 150 to 20. Does that sound ok?

It does sound like your memory tuning is completely, utterly wrong, yeah.

What you want to do is make sure that your system isn't using more memory than
it actually has.  So, work out how much memory the Apache, PHP, MySQL, and
other bits take up for each client ... then tune so you don't go beyond that.

http://www.selenic.com/smem/ is very useful to identify the actual non-shared
memory for each process, which is what the real cost comes from.

Also, http://mysqltuner.pl/ will give advice on tuning memory and other MySQL
variables based on performance and the system.


> I'd rather collect some debug info and make an evidence-based decision to
> change the maxclients option.

Good plan.  Otherwise you will have the same problem because you are still
tuned well above the real memory limit.

Other things to consider include running PHP as a FastCGI process, and using a
lighter-weight front-end server than Apache.  Collect evidence about where the
problem lies first, though, because memory tuning is hard.

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons