[Samba] Inconsistent performance (4.1.6)

[Jeremy Allison]

On Thu, Apr 10, 2014 at 11:07:25AM -0600, gabe mart wrote:
> 
> >>    socket options = IPTOS_LOWDELAY TCP_NODELAY SO_RCVBUF=131072
> >> SO_SNDBUF=131072
> 
> >Remove the above. It's pure voodoo and hasn't
> >been needed for many many years with modern kernels.
> 
> After removing the options, the performance dropped on my "good"
> machines from 80 MB/s to ~37 MB/s on reads and ~57 MB/s on writes.  On my
> "bad" machines this had no effect.

Then your kernel is severely broken :-).

> >>    aio read size = 16384
> >>    aio write size = 16384
> >>    write cache size = 268435456
> 
> >>    min receivefile size = 16384
> 
> > I get the feeling you're just adding 'performance'
> > options here without understanding what they do.
> > Especially 'write cache size' :-).
> 
> > Remove all the above and re-benchmark. Add them
> > in, one by one after reading the man pages *carefully*.
> 
> > aio read/write isn't much use without the aio_pthread
> > module for example.
> 
> 
> I must be mistaken, but aio read/write is supported by default on the compile:
> 
>   --with-aio-support
>             Build with aio-support support (default=yes)

Yes, but aio support is *completely broken* on glibc (only
allows one outstanding event per file descriptor, which is
'helpful' :-), and has been known to cause crashes on
FreeBSD.

The only safe way to use aio is to use it via the
samba-coded pthread module, which doesn't suffer
from these problems.

> Also, write cache size was input after considering the amount of cache my RAID
> controllers have and deciding on the number specified.  Although it didn't seem
> to improve performance, it did not hurt, so I left it in.  I realize that 'min

But you're not understanding what it does. It uses memory
to cache reads/writes that can cluster around 'hotspot'
areas. You only ever add this when you really know you
need it, and you understand the application profile. It's
a really dangerous thing to add in general. In the past
we have had data corruption bugs here (not saying we do
now, I wrote most of the code and I'm still confident
in it, but you really need to know what you're doing
here :-).

> Adding back in the socket options without the aio lines read performance goes
> to ~55 and write to ~40 and no affect on the "bad" machines.
> 
> Any way to rule out the performance being the samba server?

Try a raw TCP transfer from the 'bad' machines.