[Samba] This old tune again : read performance optimisations

[Jeremy Allison]

On Thu, Mar 22, 2012 at 05:49:44PM +0100, Emmanuel Florac wrote:
> Hi list,
> I have this problem of (relatively) poor samba read performance. Read
> performance is constantly, reproducibly lower than write performance
> and I can't really understand why.
> 
> Here is the server configuration :
> 
> dual octocore Opteron 6128, 32 GB RAM
> Adaptec 6445 RAID card
> 36x 2TB Hitachi SATA drives (raid 60)
> 
> Sustained local disk throughput: over 850 MB/s write, over 1.4 GB/s
> read.
> 
> kernel 3.1.10 (but I tried various versions with no more
> success), amd64, pure vanilla kernel.org version.
> Distro Debian squeeze amd64, samba  2:3.5.6~dfsg-3squeeze5
> 
> The client is an opteron dual core machine running Win7 64 bits. This is
> by far the best performing CIFS client I've tested, software wise (see
> below). 
> 
> The systems are connected in 10GigE ethernet using Myricom Myri-10G
> dual ports (only one port used) through a Fujitsu 10GigE switch.
> 
> I first tested the same client to the same server using Linux (same
> configuration as the server) and  netperf, then NFS, then cifs mount.
> 
> Netperf (TCP_SENDFILE test) gives more than 9500 Mb/s both direction, so
> the network hardware works OK.
> 
> NFS gives the following results both in UDP and TCP modes:
> Sustained throughput: 490 to 520 MB/s write, 620 to 650 MB/s read. 
> 
> The linux CIFS client running on the same machine writes at a puny 180
> MB/s and read at a miserable 75 MB/s. OK, never mind, samba is for
> windows clients, right?
> 
> Now, the same machine rebooted under windows 7 writes at 450 MB/s,
> which is close enough to NFS performance, but can't top 400 MB/s
> reading. Read performance *should* be higher. I've played with various
> parameters; I can get better write performance at times, on par with
> NFS, but I never at any time remotely get as good a read performance.
> 
> I'd really like to get this beast tamed at last :)
> 
> 
> Here comes the smb.conf file :
> 
> [global]
>     name resolve order = wins lmhosts host bcast
>     socket options = TCP_NODELAY
>     
>     interfaces = eth2
>     bind interfaces only = yes
>     aio read size = 1
>     aio write size = 1
>     aio write behind = yes

Until the next 3.6.x ships with the aio_pthread module,
you might we better off removing the "aio XXX" lines.

Linux glibc aio is fundamentally broken (and Ulrich
doesn't seem interested in fixing it) - this is why
I ended up writing the aio_pthread module.

You could try using Volker's aio_fork module, which
might help (with the Linux clients, but probably not
the Windows clients unless you're running SMB2).

You're you're more adventurous you could try running
from v3-6-test git tree and ensure you have the
aio_pthread module loaded.

Jeremy.