[RFC] Performance improvements

[Swen Schillig]

Hi Martin

As suggested by you, I added some debug messages to see how CTDB
performs under high load in the area of change.

The code is updated as discussed before
	- no patch-10 (leaving tevent immediate scheduling as is)
	- no alignment patches
	- buffer_size raised to 32k (see comment below)

As a prelimenary statement, even though the debug output is slowing
down the system it can be seen that the positive effects reported
earlier are still existing.

Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2240 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=59720 used=16 offset=59704 ready=47672 max_offset=2047816 max_ready=644680
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=80 max_offset=0 max_ready=200
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=8640 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=200 max_offset=0 max_ready=200
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2160 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3280 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1520 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2320 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=5600 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2240 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=7360 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=107464 used=64 offset=107400 ready=30608 max_offset=1906056 max_ready=682248
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3840 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3680 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3760 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=6240 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=4400 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=200 max_offset=0 max_ready=200
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3360 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=5040 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=5200 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2720 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3600 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=400 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=6240 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=7360 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1600 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=7200 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=5760 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3440 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2000 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=640 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=4560 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1840 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=7280 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1360 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3680 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2320 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1760 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=640 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2720 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=3520 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=1920 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2000 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=640 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=2480 max_offset=0 max_ready=21840
Jul 17 12:06:03 fscc-x36m3-32 ctdbd[10007]: buffer size=32768 used=0 offset=0 ready=4480 max_offset=0 max_ready=21840

as you can see there's almost always more than 1k data on the socket but only rarely more than 8k (value for "ready").
In addition we can see that the processing of the received data is almost always fully done before new data
is read from the socket (offset == 0)
I will try and put some higher load (overload ?) on the system and see if things will change.

Cheers Swen.