using virtual synchrony for CTDB

[tridge at samba.org]

Steven,

Thanks for your comments!

 > I have a suggestion to use virtual synchrony for the transport mechanism
 > of CTDB.  I think you will find using something like TCPIP unsuitable
 > for a variety of reasons.

Can you point me at any performance numbers for this? Transport
mechanisms that provide nice guarantees like message ordering and
group delivery also tend to pay a significant performance cost. I
don't expect we will need either of those features in CTDB, so I'd
very much like to avoid any performance cost that might come along
with it.

The sort of performance numbers I'm interested in are like the ones
you can get from 'netpipe', showing round trip times for varying
message sizes. For CTDB we don't need any multicast capabilities, so
that side of the performance problem isn't interesting.

 > node A B C all want to lock a resource R1 at about the same time.  They
 > all send messages A sends m_lr1A (lock resource 1), B sends m_lr1B, C
 > sends m_lrlC
 > 
 > In this case it would be possible for a variety of scenarios to occur
 > A receives m_lr1A, mlr1B, mlr1C
 > B receives m_lr1A, mlr1B, mlr1C
 > C receives m_lr1A, mlr1B, mlr1C

I need to explain a bit more in the CTDB document about locking. I
actually expect we will end up with no remote locking at all, avoiding
it by using CTDB_REQ_CONDITIONAL_APPEND calls. This changes the
characteristics of the communication rather a lot :)

 > Finally totem is available as a shared linkable library for use directly
 > in other applications.  It requires the use of the poll syscall so a
 > poll abstraction is provided with timers in this minimal configuration
 > option.

I presume it could also use epoll or select? I'd like to use epoll
where available.

Cheers, Tridge