ctdb/log.ctdb: High hopcount 578746399 dbid:0x7a19d84d key:0x1a5954bc ...

[Martin Schwenke]

Hi Rainer,

On Tue, 06 Aug 2013 14:29:10 +0200, Rainer Krienke
<krienke at uni-koblenz.de> wrote:

> I run ctdb 2.3 with samba 3.6.16 on 6 nodes. I recently upgraded from
> ctdb 1.2. I tried a smooth update. Stopping the old ctdb on one node,
> upgrading this node and starting the new version again. Then I did the
> same with the next cluster node.
> 
> Since the upgrade I noticed that ctdb is using much more CPU power. ctdb
> process is using up to 75% of one cpu depending on how many clients
> connect to a certain node. Each node has roughly the same number of
> clients. The total number of connections to the samba cluster is about
> 260 (entries counted from smbstaus -b).
> 
> In /var/log/ctdb/log.ctb I see entries like this:
> 
> High hopcount 578746399 dbid:0x7a19d84d key:0x1a5954bc pnn:0 src:3
> lmaster:0 header->dmaster:5 dst:5

These messages are meant to indicate that a request to migrate a record
is bouncing between nodes.  The usual reason for this would be
contention: the record is migrated as desired but is then migrated to
another node before it is used.

However, in this case there is an inconsistency in the database between
2 nodes that each believe the other has the current copy of the
record.  This is caused by a bug in the vacuuming code.  We think we
found and fixed the bug today after trying for a few weeks to
understand it.

If everything goes according to plan then CTDB 2.4 should be released
next week and will contain a fix for this bug.

Running "ctdb recover" should fix the inconsistency.  I haven't tried
it but I wanted to offer the suggestion before your day ends...  :-)

peace & happiness,
martin