using virtual synchrony for CTDB

[tridge at samba.org]

Steven,

 > Your test program opens two sockets one for reading and one for writing.
 > I don't know if this is your intended design, but not typical of a
 > client server application.

It's in junkcode for a reason :)

Actually, I used separate ports as it allows me to test the same code
on localhost. If you specify the same port number twice, you'll get 1
port between two hosts.

And in case you're wondering, no, this is not anything like the code
we will use in CTDB. It's just my quick and dirty "how fast is gigabit
these days?" test. In CTDB I plan on using the Samba4 events library
and epoll() where available.

 >  More likely you would see one socket for all communication (reads
 > and writes).  That is how I wrote my original app.  In this case
 > performance drops in half.

why would performance drop? This test code is fully synchronous. It
doesn't care if its one socket or two.

 > Your test program also does not test the full exchange cycle, but
 > measures ops as the number of read and write operations per second.
 > Instead, I've removed this multiplier since it doesn't really
 > measure exchanges.

yes, that's why I multiplied your result by 2 when comparing.

 > I have put it at www.broked.org:/tcp-s.c and www.broked.org:/tcp-c.c to
 > match my original app design that tested request and responses.

a few problems :-)

1) tcp-s.c has commented out the TCP_NODELAY setting. That absolutely
kills performance! It means you are sitting waiting after each send,
with the kernel hoping another send might happen on that socket so it
can combine it.

2) tcp-s.c has changed to a 64 byte buffer, but didn't change the
actual read/write size (the read and write calls are still using size
1).

 > I get about 6k exchanges in this scenario on my gige network with jumbo
 > frames amd 9000 MTU.

That's the nagle algorithm killing you. Re-enable the TCP_NODELAY and
that should rise a lot.

Cheers, Tridge