[CTDB and GFS] test: scenario and some results

[tridge at samba.org]

Ignacio,

A few comments on the ping_pong results for GFS.

First off, its great that it didn't fail! If it passed all the
ping_pong tests then thats really good news.

 > 1,2,3 nodes over the same file without options:
 > -----------------------------------------------
 > ./ping_pong  /usr/local/etc/users_1/c 4
 > 
 >   1 node:  15897 locks/sec
 >   2 nodes:  1217 locks/sec
 >   3 nodes:    33 locks/sec

The low result for 1 node is perhaps the most surprising. It involves
no contention, so GFS should be able to serve the lock locally. It is
not unexpected that it be slower than a local filesystem, but not by a
factor of 20x.

For comparison, GPFS is about 3x slower for single node ping_pong than
a local filesystem. On my test system I get 1.6M locks/sec on a local
filesystem, versus 490k locks/sec running against one node of a GPFS
cluster (I suspect my test node CPUs are a bit faster than yours).

The 1 node result means that "posix locking" will cost quite a lot for
GFS.

Relatively, the 2 node result is reasonable. It is not exactly fast,
but it's not terrible. That number matters when two clients are
contending for the same file. Again, "posix locking" will hurt if you
are serving one file to more than one client.

The 3 node result is worse than expected. You expect to lose some
performance going from 1 node to 2 nodes, but after that the drop
should be less as you add more nodes. Losing an extra factor of 40x at
3 nodes means you can't really scale beyond 2 nodes. 

 > 1,2,3 nodes over the same file with -rw options:
 > ------------------------------------------------
 > ./ping_pong -rw /usr/local/etc/users_1/c 4
 > 
 > 1 node:  13708 locks/sec
 > 2 nodes:   166 locks/sec
 > 3 nodes:    11 locks/sec

This one is interesting. The biggest drop is from 1 to 2 clients,
which means IO contention (as opposed to lock contention) kicks in at
2 nodes.  GPFS also has this property, although GPFS doesn't lose as
much when adding a 3rd node.

 > The GPFS, runs fine?

GPFS is better at some of these operations, but it still isn't nearly
as fast as I would like. I don't know of any clustered filesystem that
gets anywhere near what the hardware could potentially do for these
sorts of tests.

One of the main design aims of clustered Samba with CTDB is that we
are now much less dependent on the speed of the above operations than
we were with previous designs. So the ping_pong test must pass, but if
its slow then clustered Samba should still be usable, especially with
"posix locking = no".

btw, did the "data increment" always show the right value when you ran
the -rw test? That is an important part of the test as far as
correctness goes. 

Cheers, Tridge