[clug] Out of Memory: Kill process 2689 (mysqld) score 33827 and children.

[Edward C. Lang]

Hi,

On Wed, Aug 18, 2010 at 03:47:17PM +1000, Daniel Pittman wrote:
> Carlo Hamalainen <carlo.hamalainen at gmail.com> writes:
> > I've been watching the mod_status page of Apache and I'm not seeing
> > any processes hanging around in the 'W' state with high CPU time.
> > Usually the system goes down after 4 to 7 days so I'll have to write
> > a cron job to regularly save the Apache status page and perhaps a
> > few other things. I'm tending towards this being a plain lack of ram
> > but time will tell.
> 
> One other approach would be to install something like munin that will
> monitor resources, record their data, and allow you to review graphs
> showing metrics over time.  That might highlight where the shortage is
> coming from.

I mentioned to Carlo that I had a similar problem on my Linode a month
ago. I referenced the support ticket and the steps I took to track down,
and then correct, the problem.

I used this guide to find the cause of the high I/O wait apparent in the
vmstat output:

    http://rackerhacker.com/2008/03/11/hunting-down-elusive-sources-of-iowait/

(from memory the egrep etc commands didn't quite work as intended on my
Ubuntu 10.04 LTS install, but they're close enough)

I also used the mtop command / serverstats output to get an idea of what
happened and when. It turned out that kswapd was generating most of the
I/O, and that various large processes I wanted to keep (amavisd, Clam
AV, spamc etc) were using up most of the virtual memory.

I modified the vm.overcommit_memory and vm.overcommit_ratio values to be
``2'' and ``90'', respectively, as per the description of these values:

    http://paste.ubuntu.com/465823/

Finally, I paid for some more physical RAM. My conclusion was that Xen
(and other virtual machines) really, really hates swapping. If at all
possible, increase the size of the physical RAM.

 - Edward.

-- 

http://edlang.org/