[Samba] poor performance with bonding in round-robin mode (only samba affected)

[Ralf Gross]

Hi,

samba 3.0.24, debian etch

I'm seeing a strange effect with samba and traffic over a bond0
interface in round robin mode.

2 server each with 2 GbE interfaces as bond0 device ind rr mode.

netio benchmark:

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP connection established.
Packet size  1k bytes:  182840 KByte/s Tx,  197599 KByte/s Rx.
Packet size  2k bytes:  200947 KByte/s Tx,  208788 KByte/s Rx.
Packet size  4k bytes:  217239 KByte/s Tx,  208988 KByte/s Rx.
Packet size  8k bytes:  225145 KByte/s Tx,  208948 KByte/s Rx.
Packet size 16k bytes:  227877 KByte/s Tx,  208372 KByte/s Rx.
Packet size 32k bytes:  227802 KByte/s Tx,  208122 KByte/s Rx.
Done.

netpipe benchmark:

[...]
121: 8388605 bytes      3 times -->   1540.13 Mbps in   41554.81 usec
122: 8388608 bytes      3 times -->   1538.24 Mbps in   41605.99 usec
123: 8388611 bytes      3 times -->   1482.47 Mbps in   43171.17 usec

So the network throughput is ok in round robin mode. I know that there is
overhead because of packet reordering, but ~180MB/s is ok.


ftp (bonding mode round robin):

ftp> get 2GB_file
local: 2GB_file remote: 2GB_file
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for 2GB_file (2048000000 bytes).
226 File send OK.
2048000000 bytes received in 35.95 secs (55626.5 kB/s)

samba (bonding mode round robin):

Domain=[EMEA] OS=[Unix] Server=[Samba 3.0.24]
smb: \> get 2GB_file
getting file \2GB_file of size 2048000000 as 2GB_file (35466,7 kb/s) (average 35466,7 kb/s)

That's 20MB/s less than ftp in rr mode.


But if I change the bonding mode to xor, I get nearly the same throughput with
samba that I get with ftp.

ftp (bonding mode xor):
ftp> get 2GB_file
local: 2GB_file remote: 2GB_file
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for 2GB_file (2048000000 bytes).
226 File send OK.
2048000000 bytes received in 33.62 secs (59486.2 kB/s)

samba (bonding mode xor):
Domain=[EMEA] OS=[Unix] Server=[Samba 3.0.24]
smb: \> get 2GB_file
getting file \2GB_file of size 2048000000 as 2GB_file (52364,2 kb/s) (average 52364,2 kb/s)


What's the difference with samba and rr mode? round robin mode is working fine
with ftp and different benchmarks (dd/netpipe too). Only with samba I'm seeing
this performance drop.

I tried different socket options:

socket options = TCP_NODELAY
and 
socket options = TCP_NODELAY SO_SNDBUF=32768 SO_RCVBUF=32768 IPTOS_LOWDELA

no difference.

additionally I used the interfaces option:

interfaces = bond0 bond1

I'm confused because with round robin mode I see a better throughput in
general, only samba seems to have problems.

Ralf