[clug] Segfault error flooding my logs

[David Schoen]

Just realised at least part of what I said isn't going to work. Apache
writes a log entry once it's already generated the object and it's on
its way back to the client, in other words you won't get a log entry
for the request that is causing the segfault in apache's access log
(even though other entries are still happily finding their way in to
the log file).

If you have a content accelerator (reverse proxy) in front of apache
you may be able to use the logs from there (IIRC Squid writes the log
entry to disk at the beginning of the request). Otherwise I'd be
resorting to bad things like packet dumps, strace and luck.

Also, "[notice] child pid 12345 exit signal Segmentation fault (11)" -
I think that means this entry will be immediate.

Cheers,
Dave

2009/9/1 David Schoen <neerolyte at gmail.com>:
> I'm not sure about the terminology or the exact process, but the
> apache parent process definitely starts off a new child worker and
> forgets about the old one (not sure if it needs to or does any other
> cleaning up).
>
> I've seen _a lot_ of segfaulting apache threads, the only time it's
> ever caused an immediately noticeable problem (as in someone not
> grepping logs being able to tell it's broken) was on a Solaris box
> that was configured to collect segfault core dumps in /var/core, the
> file system fails and then normal havoc ensues.
>
> I think normally you will just get something in syslog like you've
> attached and there should be something in the apache error log about
> having lost a child when it comes back to use that proc (can't
> remember the exact error though), depending on your request volume
> this entry might not be any where near the segfault time stamp.
>
> If it's concerning* enough you should look at your relevant access log
> at the time when php was segfaulting to try and determine what
> request(s) are failing it.
>
> Cheers,
> Dave
>
> * if this box is in any way connected to the same internet I'm
> connected to, I'd be concerned.
>
> 2009/8/31 David Tulloh <david at tulloh.id.au>:
>> PCRE has numerous well known and documented segfault problems, PHP makes no
>> attempt to catch or filter these out.
>>
>> Apache forks off a new worker thread* for each request that comes in, when
>> PCRE segfaults this thread dies. Apache however carries on, so your server
>> will not go down if a few threads do. I'm not sure if these segfaulted
>> threads are cleaned up or hang around, if it's the latter your server may
>> eventually stop responding as it runs out of threads.
>>
>>
>> David
>>
>> * Assuming default Apache 2 configuration with mpm-worker model, other
>> models will vary slightly in effects.
>>
>> Ian McLeod wrote:
>>>
>>> Thanks for this..
>>>
>>> My Cacti server is still working though..
>>>
>>> Andrew Janke wrote:
>>>
>>>>
>>>> On Mon, Aug 31, 2009 at 18:36, Andrew Janke<a.janke at gmail.com> wrote:
>>>>
>>>>>
>>>>> Not that this helps but you are not alone.
>>>>>
>>>>> https://bugs.launchpad.net/ubuntu/+source/pcre3/+bug/296997
>>>>>
>>>>
>>>> And more pertinently.
>>>>
>>>> https://bugs.launchpad.net/ubuntu/+source/php5/+bug/407248
>>>>
>>>> a
>>>>
>>
>> --
>> linux mailing list
>> linux at lists.samba.org
>> https://lists.samba.org/mailman/listinfo/linux
>>
>
>
>
> --
> High Availability - You'd have to be High to think this is Available
>



-- 
High Availability - You'd have to be High to think this is Available