using virtual synchrony for CTDB

[tridge at samba.org]

Tracy,

 > Something about the DMASTER redirect is bothering me though.  Wouldn't it 
 > be possible to build up 'harmonics' in the cluster since there doesn't 
 > appear to be any terminating condition for redirects?  I've seen things 
 > like this happen.  Might be better to have a redirect limit of 1 hop or 
 > something like that and then require a return to the LMASTER.

I don't want to have a hard limit in the protocol, but I certainly do
intend that the clients will not blindly keep following redirects
forever.

Given that with version 0.2 of CTDB the LMASTER always knows who the
DMASTER is, I think a sensible rule would be that a node only follow 1
redirect before it asks the LMASTER again (ie. the 1 hop limit you
suggested). Of course, the DMASTER can move between the reply from the
LMASTER and the node trying again, so in that sense it could in theory
go on forever, but in practice I expect to not encounter more than a
couple of hops.

Part of my confidence comes from the fact that the DMASTER cannot move
until LACOUNT has reached the threshold for a move. I expect we will
set that threshold to something between 5 and 10, so in order to for a
node to get more than 1 redirect it would have to be sending messages
a full order of magnitude slower than the other nodes. That certainly
could happen for a while, but for it to be sustained indefinately
you'd have to have a pretty broken cluster.

Cheers, Tridge