[Samba] Samba Variables and TCP/IP Throughput

[AndyLiebman at aol.com]

So, here's the answer to the puzzle... I did as "Per" suggested -- set the 
so_sndbuf and so_rcvbuf to 65535 each, as well as read size and max xmit to the 
same.  And low and behold, the throughput between my Windows XP box and my 
Linux Box's Firewire RAID 5 arrays went from around 20 MB/sec to around 30 
MB/sec. And when I raised all those values by another factor of 4 (up to 262140 for 
each one) my throughput increased to around 36-37 MB/sec. Raising those values 
further didn't seem to make any difference. 

And, and, and, ...  at the 262140 level, now I was able to set the MTUs to 
4088 on the Windows side and 4074 on the Linux side (4K Jumbo Frames). While 
throughput essentially remained the same as what I was getting with a standard 
1514/1500 MTU, I was happy that it didn't DROP by 50 percent, as it did before 
when I was trying to use 4K and 9K Jumbo Frames with the buffers set to 8092.  

And although throughput didn't go UP with the 4K frames, the CPU usage in 
each machine was pretty much cut in half. That made my Windows XP video editing 
applications happy. 

Incidentally, I still get a serious DROP in throughput, about 50 percent, if 
I try to use 9K Jumbo Frames -- even with those big buffers (I even tried up 
to 1 MB buffers). Perhaps that could be due to the fact that both my Windows 
and Linux boxes only have 32-bit PCI slots. 

I should add a few other notes. First, I am using Intel Gigabit Server NICs 
because they seem to have good Linux drivers -- and the Intel Tech Support 
folks were totally blown away by the fact that a few variables in a Samba 
configuration file could affect network throughput so dramatically. Alas, Samba is a 
mystery to many folks in this world -- myself included. 

Second, after hours of playing with TCP/IP settings on both the Windows and 
Linux sides -- TcpWindowSize in Windows, and a bunch of wmem, rmem, and mem 
values in Linux -- I came to the conclusion that none of my changes gave me any 
improvement over the default configurations of both operating systems (at least 
the way Linux is configured in Mandrake 9.2). Before I started tweaking, I 
had tried all sorts of Window sizes using the program "Iperf" on both ends -- 
and found pretty much that I got to the maximum throughput (a pure bandwidth of 
about 780 Mbits) at about 64K TCPWindows. But both OS's must do that by 
default, so tweaking was a waste of time in my case, where I have a fast LAN with 
almost zero latency. 

Finally, I have to say that I am impressed with the Intel Gigabit Server 
Ethernet adapters. The Intel e1000 drivers (not the eepro1000 driver that came 
with my Mandrake 9.2 distribution) give you fantastic flexibility in both Linux 
and Windows. For instance, the driver has a number of methods to let you choose 
whether the NIC interrupts the CPU each time it receives or transmits a 
packet, or whether the NIC stores up lots of packets and interrupts less 
frequently. Playing around with that and other settings in the driver allowed me to 
trade a 15 percent reduction in maximum throughput (and a slight increase in 
latency) for a 75 percent reduction in CPU usage! Which let my Windows video 
editing application flawlessly playback uncompressed video through some hardware 
that's also vying for attention on the PCI bus.  Without the ability to do this 
kind of tweaking, my video editing application was getting interrupted too 
often to work properly. Surely there are other real time applications that can 
benefit from such control. I certainly had a happier experience with the Intel 
NIC than with either Linksys or SysKonnect NICs -- although I hear the 
SysKonnect NICs are a little faster than most. 

Hope all this information is useful to somebody out there. 

As a last word, of course my switch supports Jumbo Frames. I'm trying out a 
fairly new SMC switch, the SMC8508T. It's fast, cheap and out of control... No 
just kidding. It's fast and cheap (around $ 140 US) and as far as I can see 
it's the only switch any where near that price that supports Jumbo Frames. Says 
so right on the box. It's an unmanaged switch. But I know it's not the switch 
that's caused my original problems with Jumbo Frames, because I had the same 
problems when I took the switch out of the system and connected the computers 
directly. And that's still the case with the 9K Jumbo Frames. 

Finally, I put "use sendfile=yes" in my samba configuration file. But what 
does it do??? I suppose I should at least try taking it out and seeing what it 
does. 

Thanks for your suggestions. It's because of people like you that Linux keeps 
getting better, and Linux users keep getting better results. 

Andy

In a message dated 12/17/2003 10:10:10 AM Eastern Standard Time, 
greg at max-t.com writes:
I believe samba just does setsockopt or ioctl on the sockets. Do you get any 
errors on the interfaces in jumbo? Does your switch support jumbo? Setting 
"use sendfile=yes" will help alot on read speeds from samba. On the windows 
side check the settings. I think the e1000 has some adaptive spacing setting 
that kills throughput. Also some things to check on the linux side. e1000 
module options like rxIntDelay, etc.

You will not get much more performance out of jumbo unless your CPUs are 
maxed 
but you should not get less. What kind of numbers are you seeing? 


hope this helps.
Greg


On Wednesday 17 December 2003 08:23 am, AndyLiebman at aol.com wrote:
> Thanks for the reply. Do you know (and if so, caan you tell me) what the
> relationship is between these Samba settings and Linux settings such as
> net.core.rmem_default (or _max), net.core.wmem_default (or _max),
> net.ipv4.tcp_rmem and net.ipv4.tcp_wmem.  Do the Samba options override the
> Linux socket options, or do they act as another layer of limits and
> buffers?
>
> Perhaps your TCP window is too small
> You should try the following global settings:
>
>     read size = 65535
>     max xmit = 65535
>     socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535
>
>
> Rgds Per
>
> AndyLiebman at aol.com wrote:
> > Hi,
> >
> > I am trying to optimize my gigabit network. I have two Intel 1000 MT
> > Gigabit Server Adapters, which support Jumbo Frames -- as well as a
> > Switch that supports Jumbo Frames. However, I am observing some strange
> > behavior in my
>
> file
>
> > transfers from Windows XP to Linux and I am wondering if it has anything
> > to
>
> do
>
> > with the way the Samba variables are set on my Linux box?
> >
> > The "strange behavior" is that when I set both NICs to use Jumbo Frames
> > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux
> > side (doesn't include the headers], I am getting about half the
> > throughput that
>
> I get
>
> > when I set both NICs to use the standard MTU of 1514/1500. I see the same
> > behavior even if I take the switch out of the system and connect the
>
> Windows XP and
>
> > Linux machines directly to each other (crossover cable not required for
> > computer-to-computer connection with these NICs -- and by the way all of
> > my
>
> cables
>
> > are CAT6).
> >
> > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all
> > of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The
> > Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in
> > hyperthreading mode.
> >
> > I am wondering if any of the Samba socket options settings like
> > tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput
> > -- particularly when I am using Jumbo Frames? And are there any other
> > Samba settings that
>
> might
>
> > be interacting in a negative way with my TCP/IP and NIC driver settings
> > that are causing me to get lower throughput with Jumbo Frames instead of
> > higher throughput (which is what I am told I should be getting).
> >
> > Any guidance would be appreciated. I have purchased "The Official Samba 3
> > HOW-TO and Reference Guide" but it really isn't very helpful when it
> > comes
>
> to
>
> > understanding how to tune these options and how various socket options
>
> settings
>
> > interact with other network settings and hardware.
> >
> > Andy Liebman
> > Resolute Films
> > 119 Braintree Street, Suite 410
> > Boston, MA 02134
> >
> > Tel: 617-782-0479
> > Cell: 617-308-0488
> > Fax: 617-782-1071
> > --
> > To unsubscribe from this list go to the following URL and read the
> > instructions:  http://lists.samba.org/mailman/listinfo/samba