using virtual synchrony for CTDB

[tridge at samba.org]

Steven,

 > In raw throughput terms, vs can deliver approx 40k messages per second
 > and does indeed saturate a GIGe network, whereas (over a switched
 > network) point to point can handle 1600 exchanges per second between two
 > nodes.  This scales to multiples of 1600 between multiple nodes
 > depending on switch bandwidth availability.

Those numbers don't look good I'm afraid. Between two nodes I measured
31000 messages/sec on gigabit (both with TCP and UDP). With some
fancier hardware and MPI we measured around 170000 messages/sec with
netpipe.

Maybe you could try the test code at
http://samba.org/ftp/unpacked/junkcode/messaging/ on your nodes? I'd
be curious how close you get to the 31000 we measured.

To use the tcp2 example, compile on two hosts and run:

on host1:  ./tcp2 host2 2000 2001
on host2:  ./tcp2 host1 2001 2002

it's very primitive code, but is useful for basic throughput on tcp
and udp. Also does unix domain sockets.

Note that I get about 17000 messages/sec on my 100 MBit switch at home
between my laptop and a server box. That uses a 1 byte payload. With a
64 byte payload I get about 14000 messages/sec on 100 MBit ethernet.

So 1600 exchanges/sec (presumably 3200 messages/sec for coomparison
with my test) is very slow for gigabit.

Cheers, Tridge