Further information on sendfile under FreeBSD

[Richard Sharpe]

Further to my earlier post, here is data on read throughput for one client 
to 20 clients, with and without sendfile. The tests were done via the 
loopback device, which is less than satisfactory, but I don't have the 
equipment to set up two machines with multiple GigE connectors between 
them.

The test involved putting 20 50MB files on the server, then making sure 
they were in the cache (reading them in once) and then running a test to 
read the whole file, with and without oplocks requested. Requesting 
OpLocks allows the sendfile code in Samba to kick in.

Here are the results:

Clients       MB/S read w/o OpLocks    MB/S read with OpLocks
   1                  171.21 MB/s              171.22 MB/s 
   2                  151.09 MB/s              165.54 MB/s
   3                  125.24 MB/s              158.78 MB/s
   4                  127.59 MB/s              159.30 MB/s
   5                  121.17 MB/s              152.00 MB/s
   6                  113.33 MB/s              149.59 MB/s
   7                  113.43 MB/s              149.80 MB/s
   8                  111.97 MB/s              146.23 MB/s
   9                  112.21 MB/s              145.83 MB/s
  10                  111.69 MB/s              146.37 MB/s
  11                  110.99 MB/s              145.63 MB/s
  12                  111.27 MB/s              144.70 MB/s
  13                  110.57 MB/s              145.49 MB/s
  14                  111.33 MB/s              144.11 MB/s
  15                  111.85 MB/s              144.03 MB/s
  16                  111.23 MB/s              145.22 MB/s
  17                  111.11 MB/s              143.43 MB/s
  18                  111.08 MB/s              145.16 MB/s
  19                  111.55 MB/s              143.74 MB/s
  20                  111.31 MB/s              144.10 MB/s

Pretty clearly, the CPU is saturated from the outset. Here sendfile is 
helping up get higher bandwidth with larger numbers of clients. Other data 
I have for slower CPUs show that we can go close to saturating GigE with 
sendfile and a lot lower than that without. The above data suggests that 
this machine might be able to saturate two GigE adapters while reading 
from files in the cache.

Regards
-----
Richard Sharpe, rsharpe at ns.aus.com, rsharpe at samba.org, 
sharpe at ethereal.com