ctdbd error out 'ctdb_send_dmaster_reply unable to update dmaster'

[ronnie sahlberg]

which version of ctdb and samba are you using?

This looks like your TDB database got corrupted.


regards
ronnie sahlberg


On Fri, Jul 1, 2011 at 1:12 PM, Yu Liao <liaoyu17 at gmail.com> wrote:
> No, the CPU is not 100%, and there is enough diskspace.
>
> There are some bad magic error from log.ctdb
>
> 2011/06/13 13:18:39.335265 [vacuum-locking.tdb:30116]:
> server/ctdb_vacuum.c:260 Traverse error in vacuuming 'locking.tdb'
> 2011/06/13 13:18:39.335278 [vacuum-locking.tdb:30116]:
> server/ctdb_vacuum.c:652 Failed to vacuum 'locking.tdb'
> 2011/06/13 13:23:34.143771 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143818 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143833 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143845 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143858 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143871 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143885 [28003]: common/ctdb_ltdb.c:168 Failed to store
> dynamic data
> 2011/06/13 13:23:34.143899 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143911 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143923 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143935 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143948 [28003]: /var/lib/ctdb/locking.tdb.0:tdb_rec_read
> bad magic 0x0 at offset=1111638594
> 2011/06/13 13:23:34.143960 [28003]: common/ctdb_ltdb.c:168 Failed to store
> dynamic data
> 2011/06/13 13:23:34.143971 [28003]: ctdb fatal error:
> ctdb_send_dmaster_reply unable to update dmaster
> 2011/06/13 13:23:34.143996 [28003]:
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> 2011/06/13 13:23:34.144010 [28003]: INTERNAL ERROR: Signal 6 in ctdbd pid
> 280032011/06/13 13:23:34.144020 [28003]:
> Please read the file BUGS.txt in the distribution
> 2011/06/13 13:23:34.144030 [28003]:
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
> 2011/7/1 ronnie sahlberg <ronniesahlberg at gmail.com>
>>
>> That error is caused when a call to
>> tdb_store() has failed with an error.
>>
>> Did you run out of memory? or diskspace?
>>
>> regards
>> ronnie s
>>
>>
>> On Fri, Jul 1, 2011 at 12:31 PM, Yu Liao <liaoyu17 at gmail.com> wrote:
>> > Hi,
>> >
>> > I configure the samba server under ctdb mode, it works fine for couple
>> > of
>> > days.  Yesterday, the ctdbd is crashed and error out 'ctdb fatal error:
>> > ctdb_send_dmaster_reply unable to update dmaster'. What is the possible
>> > reason of the error?
>> >
>> > 2011/06/13 13:23:34.143960 [28003]: common/ctdb_ltdb.c:168 Failed to
>> > store
>> > dynamic data
>> > 2011/06/13 13:23:34.143971 [28003]: ctdb fatal error:
>> > ctdb_send_dmaster_reply unable to update dmaster
>> > 2011/06/13 13:23:34.143996 [28003]: =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> > =-=
>> > 2011/06/13 13:23:34.144010 [28003]: INTERNAL ERROR: Signal 6 in ctdbd
>> > pid
>> > 280032011/06/13 13:23:34.144020 [28003]:
>> > Please read the file BUGS.txt in the distribution
>> > 2011/06/13 13:23:34.144030 [28003]:
>> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> > 2011/06/13 13:23:34.144041 [28003]: PANIC: internal error
>> > 2011/06/13 13:23:34.144348 [28003]: BACKTRACE: 25 stack frames:
>> >  #0 /usr/sbin/ctdbd [0x455800]
>> >  #1 /usr/sbin/ctdbd [0x455ac9]
>> >  #2 /usr/sbin/ctdbd [0x455bf0]
>> >  #3 /usr/sbin/ctdbd [0x455c25]
>> >  #4 /lib64/libc.so.6 [0x2ad4a20e4fc0]
>> >  #5 /lib64/libc.so.6(gsignal+0x35) [0x2ad4a20e4f45]
>> >  #6 /lib64/libc.so.6(abort+0x110) [0x2ad4a20e6340]
>> >  #7 /usr/sbin/ctdbd [0x447895]
>> >  #8 /usr/sbin/ctdbd [0x421ed6]
>> >  #9 /usr/sbin/ctdbd [0x422105]
>> >  #10 /usr/sbin/ctdbd [0x4232e5]
>> >  #11 /usr/sbin/ctdbd [0x41cf76]
>> >  #12 /usr/sbin/ctdbd [0x4034ca]
>> >  #13 /usr/sbin/ctdbd [0x46f382]
>> >  #14 /usr/sbin/ctdbd [0x44664b]
>> >  #15 /usr/sbin/ctdbd [0x446d19]
>> >  #16 /usr/sbin/ctdbd [0x46c569]
>> >  #17 /usr/sbin/ctdbd [0x46cc2e]
>> >  #18 /usr/sbin/ctdbd [0x46868e]
>> >  #19 /usr/sbin/ctdbd [0x4688b3]
>> >  #20 /usr/sbin/ctdbd [0x46896b]
>> >  #21 /usr/sbin/ctdbd [0x406a9e]
>> >  #22 /usr/sbin/ctdbd [0x404779]
>> >  #23 /lib64/libc.so.6(__libc_start_main+0xf4) [0x2ad4a20d2304]
>> >  #24 /usr/sbin/ctdbd [0x4033a9]
>> >
>> > --
>> > Best Regards.
>> > Yu Liao
>> >
>
>
>
> --
> Best Regards.
> Yu Liao
>