2.5 readpages doubles cifs vfs large file copy performance vs. 2.4

[Steven French]

Interesting data today.  We did a simple but useful test - single client
large file copy Linux to Linux on Gigabit Ethernet using various network
filesystems clients.   NFS as expected does a better job at keeping
multiple read requests on the wire at the same time, so performs about six
times faster than cifs vfs on 2.4 (I tried this on 2.4.20 RedHat adding a
cifs.o module built here but the results should be similar to the what
someone would get with the cifs vfs that is already included in SuSE and
other UL distributions since the cifs.o included in SuSE is only moderately
backlevel).  NFS performance on Linux to Linux was reasonably close to
Gigabit ethernet maximums.  For the cifs client as expected things look
much better on 2.5.  Keeping the server constant and rebooting the client
to 2.5 - for the cifs vfs included in 2.5, readpages helps a lot (more than
doubles the performance 2.5 vs. 2.4 cifs vfs).   nfs client seemed a bit
slower on 2.5 than nfs on 2.4 for this but not dramatically slower.   nfs
was mounted with nfs v3 using UDP, don't know what it would look like for
tcp, presumably a bit slower but still faster than cifs vfs on 2.5 until I
make some performance improvements.

On cifs vfs on 2.4 traces show a lot of surprisingly long delays between
the data getting sent from the server and the client acking the final read
response data and between the last ack and the client sending the next read
request, but fortunately on 2.5 there are 1/4 the cifs read requests than
2.4 due to ability to handle readpages on Linux 2.5

Since the disk subsystems are not really fast enough, the files on the
server side are cached to get results this good.   Interestingly Samba
server itself (vs. nfsd) did not seem to be a big problem from the
performance perspective but rather the main difference for this kind of
test is just having the server not sitting idle waiting to receive a new
read request (unlike cifs client, nfs client issues more than one read
request at a time which helps there).

But what was most interesting to me was thinking whether a simple mod to
cifs_readpages and CIFSSMBRead to send about three requests at a time (even
though Samba server may end up processing them synchronously) is all that
is required to also bring cifs vfs reasonably close to optimal GigE single
file read performance without having to adopt the very complex approach to
readahead and page management used by the Linux nfs client.

We also need to retry this with Samba 3.0 (this was against Redhat's
version of Samba 2.2) but the server appears to not contribute as much as
we expected to the delay in relatively unstressed file copy.

Steve French
Senior Software Engineer
Linux Technology Center - IBM Austin
phone: 512-838-2294
email: sfrench at-sign us dot ibm dot com