[Performance] Samba 3 vs. Samba 4 performance in NetBench

[Andrew Tridgell]

Marc,

Thank you for sending me the sniffs comparing Samba3 and Samba4 in
netbench. They are certainly interesting!

The differences I have noticed so far are:

1) samba4 negotiates a buffer size 12288 vs 16644 for Samba3

2) you have Samba3 setup in share level security, Samba4 in user level
   (probably not significant)

3) The test files already exist with samba3, not with samba4. Maybe
   you started the runs in different ways? This shouldn't matter, but
   I'm curious as to how it happened given the way netbench works.

4) samba3 reports 1 sector per unit in QUERY_FS_INFO, Samba3 reports 2

5) the read sizes between the two runs are about the same, but the
   write sizes are vastly different!


The last point is probably the key factor. If we look at a histogram
of write sizes for Samba3 and Samba4 we see:

Samba3:
    1378 (3%) 4
    1490 (3%) 4096
    2239 (4%) 88
    2452 (5%) 16384
    2506 (5%) 65536
    3422 (6%) 2
    3490 (7%) 65534
    3778 (7%) 1
    8190 (15%) 2048
    11634 (22%) 512

Samba4:
    946 (1%) 4096
    1050 (1%) 4
    1728 (2%) 88
    1858 (2%) 65536
    1877 (2%) 16384
    2666 (3%) 65534
    2735 (3%) 2
    6479 (7%) 2048
    8867 (10%) 512
    49509 (56%) 1

I have omitted tail ends of the histograms.
    
So this means that for some reason your client is doing a massive
number (56%) of 1 byte write calls with Samba4, but only a moderate
number (7%) for Samba3. The trick will be to work out why!

My guess is this is caused by the extreme rounding of the allocation
size in Samba3 QUERY_FILE_INFO calls. Samba3 rounds up to the nearest
1MB, whereas Samba4 rounds to the nearest 512 bytes. In the sniffs I
see the client doing long series of 1 byte writes at gaps of about
2k. It doesn't write the data in between, it just does things like
this:

    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 4607
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 8703
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 12799
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 16895
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 20991
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 25087
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 29183
    SMB Write AndX Request, FID: 0x0200, 1 byte at offset 33279

going on for hundreds of operations. I'm guessing the client is trying
to force allocation of disk blocks. For Samba3 is skips these, as it
thinks it only has to do one write every 1M to force allocation (in
fact, Samba3 is lying and it does really need to do a write every 4k
or so to force allocation, but the client doesn't know that).

I've now added a tuning parameter in Samba4 for this. You can do:

  posix:allocationrounding = 0x100000

and it will round to the nearest 1M (same as Samba3). 

If you are using xattrs in Samba4 then it instead uses the allocation
information as provided by the client, but for the setup you are
testing (not using xattrs for both Samba3 and Samba4) the above change
should make things match better. I'll be interested to hear what
difference it makes. Note you will need svn version 7793 or later to
get the above option.

To test its working, try putting a small file on a samba4 share, and
using the allinfo command in smbclient to see what the allocation size
is being rounded to. 

All this is really just a benchmark hack btw, but it will be
interesting to see what effect it has.

Cheers, Tridge