[Samba] Samba Variables and TCP/IP Throughput
AndyLiebman at aol.com
AndyLiebman at aol.com
Thu Dec 18 04:06:24 GMT 2003
So, here's the answer to the puzzle... I did as "Per" suggested -- set the
so_sndbuf and so_rcvbuf to 65535 each, as well as read size and max xmit to the
same. And low and behold, the throughput between my Windows XP box and my
Linux Box's Firewire RAID 5 arrays went from around 20 MB/sec to around 30
MB/sec. And when I raised all those values by another factor of 4 (up to 262140 for
each one) my throughput increased to around 36-37 MB/sec. Raising those values
further didn't seem to make any difference.
And, and, and, ... at the 262140 level, now I was able to set the MTUs to
4088 on the Windows side and 4074 on the Linux side (4K Jumbo Frames). While
throughput essentially remained the same as what I was getting with a standard
1514/1500 MTU, I was happy that it didn't DROP by 50 percent, as it did before
when I was trying to use 4K and 9K Jumbo Frames with the buffers set to 8092.
And although throughput didn't go UP with the 4K frames, the CPU usage in
each machine was pretty much cut in half. That made my Windows XP video editing
Incidentally, I still get a serious DROP in throughput, about 50 percent, if
I try to use 9K Jumbo Frames -- even with those big buffers (I even tried up
to 1 MB buffers). Perhaps that could be due to the fact that both my Windows
and Linux boxes only have 32-bit PCI slots.
I should add a few other notes. First, I am using Intel Gigabit Server NICs
because they seem to have good Linux drivers -- and the Intel Tech Support
folks were totally blown away by the fact that a few variables in a Samba
configuration file could affect network throughput so dramatically. Alas, Samba is a
mystery to many folks in this world -- myself included.
Second, after hours of playing with TCP/IP settings on both the Windows and
Linux sides -- TcpWindowSize in Windows, and a bunch of wmem, rmem, and mem
values in Linux -- I came to the conclusion that none of my changes gave me any
improvement over the default configurations of both operating systems (at least
the way Linux is configured in Mandrake 9.2). Before I started tweaking, I
had tried all sorts of Window sizes using the program "Iperf" on both ends --
and found pretty much that I got to the maximum throughput (a pure bandwidth of
about 780 Mbits) at about 64K TCPWindows. But both OS's must do that by
default, so tweaking was a waste of time in my case, where I have a fast LAN with
almost zero latency.
Finally, I have to say that I am impressed with the Intel Gigabit Server
Ethernet adapters. The Intel e1000 drivers (not the eepro1000 driver that came
with my Mandrake 9.2 distribution) give you fantastic flexibility in both Linux
and Windows. For instance, the driver has a number of methods to let you choose
whether the NIC interrupts the CPU each time it receives or transmits a
packet, or whether the NIC stores up lots of packets and interrupts less
frequently. Playing around with that and other settings in the driver allowed me to
trade a 15 percent reduction in maximum throughput (and a slight increase in
latency) for a 75 percent reduction in CPU usage! Which let my Windows video
editing application flawlessly playback uncompressed video through some hardware
that's also vying for attention on the PCI bus. Without the ability to do this
kind of tweaking, my video editing application was getting interrupted too
often to work properly. Surely there are other real time applications that can
benefit from such control. I certainly had a happier experience with the Intel
NIC than with either Linksys or SysKonnect NICs -- although I hear the
SysKonnect NICs are a little faster than most.
Hope all this information is useful to somebody out there.
As a last word, of course my switch supports Jumbo Frames. I'm trying out a
fairly new SMC switch, the SMC8508T. It's fast, cheap and out of control... No
just kidding. It's fast and cheap (around $ 140 US) and as far as I can see
it's the only switch any where near that price that supports Jumbo Frames. Says
so right on the box. It's an unmanaged switch. But I know it's not the switch
that's caused my original problems with Jumbo Frames, because I had the same
problems when I took the switch out of the system and connected the computers
directly. And that's still the case with the 9K Jumbo Frames.
Finally, I put "use sendfile=yes" in my samba configuration file. But what
does it do??? I suppose I should at least try taking it out and seeing what it
Thanks for your suggestions. It's because of people like you that Linux keeps
getting better, and Linux users keep getting better results.
In a message dated 12/17/2003 10:10:10 AM Eastern Standard Time,
greg at max-t.com writes:
I believe samba just does setsockopt or ioctl on the sockets. Do you get any
errors on the interfaces in jumbo? Does your switch support jumbo? Setting
"use sendfile=yes" will help alot on read speeds from samba. On the windows
side check the settings. I think the e1000 has some adaptive spacing setting
that kills throughput. Also some things to check on the linux side. e1000
module options like rxIntDelay, etc.
You will not get much more performance out of jumbo unless your CPUs are
but you should not get less. What kind of numbers are you seeing?
hope this helps.
On Wednesday 17 December 2003 08:23 am, AndyLiebman at aol.com wrote:
> Thanks for the reply. Do you know (and if so, caan you tell me) what the
> relationship is between these Samba settings and Linux settings such as
> net.core.rmem_default (or _max), net.core.wmem_default (or _max),
> net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Do the Samba options override the
> Linux socket options, or do they act as another layer of limits and
> Perhaps your TCP window is too small
> You should try the following global settings:
> read size = 65535
> max xmit = 65535
> socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535
> Rgds Per
> AndyLiebman at aol.com wrote:
> > Hi,
> > I am trying to optimize my gigabit network. I have two Intel 1000 MT
> > Gigabit Server Adapters, which support Jumbo Frames -- as well as a
> > Switch that supports Jumbo Frames. However, I am observing some strange
> > behavior in my
> > transfers from Windows XP to Linux and I am wondering if it has anything
> > to
> > with the way the Samba variables are set on my Linux box?
> > The "strange behavior" is that when I set both NICs to use Jumbo Frames
> > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux
> > side (doesn't include the headers], I am getting about half the
> > throughput that
> I get
> > when I set both NICs to use the standard MTU of 1514/1500. I see the same
> > behavior even if I take the switch out of the system and connect the
> Windows XP and
> > Linux machines directly to each other (crossover cable not required for
> > computer-to-computer connection with these NICs -- and by the way all of
> > my
> > are CAT6).
> > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all
> > of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The
> > Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in
> > hyperthreading mode.
> > I am wondering if any of the Samba socket options settings like
> > tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput
> > -- particularly when I am using Jumbo Frames? And are there any other
> > Samba settings that
> > be interacting in a negative way with my TCP/IP and NIC driver settings
> > that are causing me to get lower throughput with Jumbo Frames instead of
> > higher throughput (which is what I am told I should be getting).
> > Any guidance would be appreciated. I have purchased "The Official Samba 3
> > HOW-TO and Reference Guide" but it really isn't very helpful when it
> > comes
> > understanding how to tune these options and how various socket options
> > interact with other network settings and hardware.
> > Andy Liebman
> > Resolute Films
> > 119 Braintree Street, Suite 410
> > Boston, MA 02134
> > Tel: 617-782-0479
> > Cell: 617-308-0488
> > Fax: 617-782-1071
> > --
> > To unsubscribe from this list go to the following URL and read the
> > instructions: http://lists.samba.org/mailman/listinfo/samba
More information about the samba