[Performance] Samba 3 vs. Samba 4 performance in NetBench

[Kaplan, Marc]

Tridge:

Looks like the NetBench hack did indeed bring the performance of Samba 4
up to near parity with Samba 3 (no oplocks). I am using the exact same
config as last time as far as the servers and clients go, the only
changes to the setup are that I used Samba4-SVN-7804 and I set
posix:allocationrounding = 0x100000 in smb.conf. 

Here are the results:
NETBENCH
Clients 	Samba3-nooplocks	Samba4-normal	Samba4-1M alloc
 1		51.150		42.285		43.776
 4		203.449		171.702		194.142
 8		204.407		170.333		193.482
 12		194.383		159.905		181.612
 16		189.470		154.434		175.575
 20		185.257		150.425		171.574
 24		185.442		149.986		170.001
 28		184.451		149.125		172.312
 32		182.569		150.044		173.536

So right now Samba 4 with the alloc hack is consistently 10Mbit/s slower
than samba3. If you think that a network trace when samba has
posix:allocationrounding = 0x100000 would be insightful let me know and
I'll upload it to samba.org.

Also: I did the win2k3 single client NetBench sniff. You can find it in
/home/mkaplan/win2k3-netbench-1client.cap.rz. It's about 50 some Megs
compressed and 2.1GB uncompressed.
	
			-Marc
> -----Original Message-----
> From: Andrew Tridgell [mailto:tridge at osdl.org]
> Sent: Monday, June 20, 2005 9:45 PM
> To: Kaplan, Marc
> Cc: samba-technical at lists.samba.org
> Subject: RE: [Performance] Samba 3 vs. Samba 4 performance in NetBench
> 
> Marc,
> 
> Thank you for sending me the sniffs comparing Samba3 and Samba4 in
> netbench. They are certainly interesting!
> 
> The differences I have noticed so far are:
> 
> 1) samba4 negotiates a buffer size 12288 vs 16644 for Samba3
> 
> 2) you have Samba3 setup in share level security, Samba4 in user level
>    (probably not significant)
> 
> 3) The test files already exist with samba3, not with samba4. Maybe
>    you started the runs in different ways? This shouldn't matter, but
>    I'm curious as to how it happened given the way netbench works.
> 
> 4) samba3 reports 1 sector per unit in QUERY_FS_INFO, Samba3 reports 2
> 
> 5) the read sizes between the two runs are about the same, but the
>    write sizes are vastly different!
> 
> 
> The last point is probably the key factor. If we look at a histogram
> of write sizes for Samba3 and Samba4 we see:
> 
> Samba3:
>     1378 (3%) 4
>     1490 (3%) 4096
>     2239 (4%) 88
>     2452 (5%) 16384
>     2506 (5%) 65536
>     3422 (6%) 2
>     3490 (7%) 65534
>     3778 (7%) 1
>     8190 (15%) 2048
>     11634 (22%) 512
> 
> Samba4:
>     946 (1%) 4096
>     1050 (1%) 4
>     1728 (2%) 88
>     1858 (2%) 65536
>     1877 (2%) 16384
>     2666 (3%) 65534
>     2735 (3%) 2
>     6479 (7%) 2048
>     8867 (10%) 512
>     49509 (56%) 1
> 
> I have omitted tail ends of the histograms.
> 
> So this means that for some reason your client is doing a massive
> number (56%) of 1 byte write calls with Samba4, but only a moderate
> number (7%) for Samba3. The trick will be to work out why!
> 
> My guess is this is caused by the extreme rounding of the allocation
> size in Samba3 QUERY_FILE_INFO calls. Samba3 rounds up to the nearest
> 1MB, whereas Samba4 rounds to the nearest 512 bytes. In the sniffs I
> see the client doing long series of 1 byte writes at gaps of about
> 2k. It doesn't write the data in between, it just does things like
> this:
> 
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 4607
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 8703
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 12799
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 16895
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 20991
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 25087
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 29183
>     SMB Write AndX Request, FID: 0x0200, 1 byte at offset 33279
> 
> going on for hundreds of operations. I'm guessing the client is trying
> to force allocation of disk blocks. For Samba3 is skips these, as it
> thinks it only has to do one write every 1M to force allocation (in
> fact, Samba3 is lying and it does really need to do a write every 4k
> or so to force allocation, but the client doesn't know that).
> 
> I've now added a tuning parameter in Samba4 for this. You can do:
> 
>   posix:allocationrounding = 0x100000
> 
> and it will round to the nearest 1M (same as Samba3).
> 
> If you are using xattrs in Samba4 then it instead uses the allocation
> information as provided by the client, but for the setup you are
> testing (not using xattrs for both Samba3 and Samba4) the above change
> should make things match better. I'll be interested to hear what
> difference it makes. Note you will need svn version 7793 or later to
> get the above option.
> 
> To test its working, try putting a small file on a samba4 share, and
> using the allinfo command in smbclient to see what the allocation size
> is being rounded to.
> 
> All this is really just a benchmark hack btw, but it will be
> interesting to see what effect it has.
> 
> Cheers, Tridge