[clug] Out of Memory: Kill process 2689 (mysqld) score 33827 and children.

[Daniel Pittman]

David Schoen <dave at lyte.id.au> writes:
> On 17 August 2010 12:17, Daniel Pittman <daniel at rimspace.net> wrote:
>>
>> Carlo Hamalainen <carlo.hamalainen at gmail.com> writes:
>>
>> > I have a 512Mb Linode server running Ubuntu 8.10 which has been slowly
>> > apt-get upgraded towards 9.04. The server runs a few Wordpress blogs
>> > and a satchmo/django shop. Every week or so cpu usage goes to 100%,
>> > everything becomes unresponsive, and the console shows this:
>> >
>> > Out of Memory: Kill process 2961 (apache2) score 44591 andchildren.
>> > Out of memory: Killed process 3019 (apache2).
>> > Out of Memory: Kill process 2689 (mysqld) score 33827 and children.
>> > Out of memory: Killed process 2689 (mysqld).
>> > Out of Memory: Kill process 2699 (mysqld) score 33827 and children.
>> > Out of memory: Killed process 2699 (mysqld).
>> > Out of Memory: Kill process 2703 (mysqld) score 33827 and children.
>> > Out of memory: Killed process 2703 (mysqld).
>> > Out of Memory: Kill process 2961 (apache2) score 21690 and children.
>> > Out of memory: Killed process 3022 (apache2).
>> > Out of Memory: Kill process 15914 (mysqld) score 34048 and children.
>> > Out of memory: Killed process 15914 (mysqld).
>> >
>> > I have to hard reset the server as ssh is completely unresponsive.
>> >
>> > So, is this really a memory leak in mysqld?
>
> I doubt it. The failure case I normally see is that *something* is
> causing deadlock on a DB resource and Apache doesn't have a low enough
> MaxClients configured.

Do you really think so?  With 150 Apache clients, running PHP, I would be
pretty surprised if this wasn't just regular old memory exhaustion.  They only
have 3MB of unique memory per Apache process *without* counting anything like
the MySQL service, or the kernel, or anything else.

[...]

>> What you want to do is make sure that your system isn't using more memory than
>> it actually has.  So, work out how much memory the Apache, PHP, MySQL, and
>> other bits take up for each client ... then tune so you don't go beyond that.
>>
>> http://www.selenic.com/smem/ is very useful to identify the actual non-shared
>> memory for each process, which is what the real cost comes from.
>>
>> Also, http://mysqltuner.pl/ will give advice on tuning memory and other MySQL
>> variables based on performance and the system.
>
> Tuning is always a good thing, but if it's a deadlock issue it will
> not solve your current issue.

Yeah, I just have trouble believing that there is anything particularly likely
to involve a deadlock in this scenario.


Actually, I am surprised you see so many deadlocks in MySQL, frankly.  Pretty
much none of the PHP applications I am familiar with do any locking at all,
just random updates without transactions.  Which packages usually trigger this
sort of lock problem?

[...]

> For reference some of the issues I've resolved like this were related to
> combinations of flakey components. I've seen bugs in both eAccelerator and
> APC terminate threads in bad ways so that resources on the FS could never be
> released, causing all new threads to queue up.

Interesting.  Were those resources flocks on their cache files, or something
else?  Which versions were they found / resolved in?  (Bonus points for links
to your upstream bug reports, since they will have all those details without
you having to repeat them. ;)

> I've seen complex php pages terminate early due to a low a php memory_limit
> that would leave their DB connection dangling and cause new threads to
> deadlock (that one only happened with eAccelerator enabled though). I've
> seen maintenance scripts that lock FS objects completely separately to
> either httpd or postgres (I don't use mysql) and cause all new threads to
> queue up on those locks. All of these were resolved with server status,
> strace, lsof and DB debugging options in the logs.

*nod*  Very strange.  It sounds like a whole lot of the problems come from the
range of crappy PHP "accelerators" that are intended to work around the nasty,
non-persistent nature of PHP.  Is that a fair summary?

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons