[Samba] SMB/CIFS seq. transfers top out at 30MiB/s (NFSv4 and HTTP: 100MiB/s+)

[Jeff Layton]

On Wed, 20 Jan 2010 19:41:24 +0100
Johannes Truschnigg <johannes at truschnigg.info> wrote:

> Hello list,
> 
> I'm using Samba 3.4.5 on a home-hosted fileserver of mine to easily share 
> files with both GNU/Linux and Windows XP (Professional 32Bit SP3) clients. The 
> machines are connected to each other via a switched GBit ethernet network, the 
> actual available bandwidth between the server and the nodes over TCP amounts 
> to about 940-980MBit (according to iperf).
> 
> The server's storage backend is aquite potent, and achieves sequential read 
> and write speeds well over the network's linespeed (somewhere around 180MiB/s 
> read and 130MiB/s write at worst and well over 200MiB/s and 160MiB/s, resp., 
> at best). When serving files from the very same filesystem as with Samba, 
> Apache 2.2 delivers about 110MiB/s on average, and NFSv4-transfers match that 
> number in terms of speed.
> 
> With Samba and SMB/CIFS, however, the transfer speed tops out at rather 
> disappointing 28-30MiB/s, serving both Windows and GNU/Linux clients. I'm 
> using the in-kernel CIFS support on the GNU/Linux machines, and the default 
> "Attach Network Drive"-feature with Windows. Using the (afaik pure userspace) 
> `smbclient` implementation on the GNU machine doesn't change anything to the 
> better, the speed remains at the aforementioned ~30MiB/s.
> 
> The server system isn't really loaded while serving files over CIFS (its cores 
> don't even clock to higher frequencies, but remain at a comfortable 1GHz), and 
> smbd never consumes substantially more than ~15% CPUtime while reading from 
> disk and delivering to the clients.
> 
> I tried adopting advice from the Samba manual's performance tuning section 
> (http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/speed.html), but 
> that didn't improve the situation at all - transfer speed remained roughly the 
> same, compared to these settings' default values. Playing with the clients' 
> mount options (rsize and wsize, specifically) didn't yield any noticeable 
> improvements, either.
> 
> I continue to see other peoples' reports on the web that they manage to 
> squeeze much higher transfer rates (70MiB/s+) out of dedicated NAS appliances 
> via SMB/CIFS (most of which supposedly also run Samba for providing that 
> service), and I fail to see why my platform won't deliver similar results, as 
> the hardware should easily match whatever those NAS-devices offer.
> 
> Below I will list what I think might be relevant information to track down 
> what's wrong; in case I'm missing something that'd be of use, please, let me 
> know!
> 
> ----SNIP START: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf ----
> [global]
>     workgroup = ARBEITSGRUPPE
>     security = share
>     load printers = no
>     guest account = nobody
>     dns proxy = no
>     syslog = 1
>     syslog only = yes
>     socket options = SO_RCVBUF=8192 SO_SNDBUF=8192
>     disable netbios = yes
>     unix extensions = yes
>     unix charset = UTF-8
>     display charset = UTF-8
>     use mmap = yes
>     use sendfile = yes
>     wins support = no
> [files]
>    path = /srv/files/pub/
>    public = yes
>    only guest = yes
>    writable = yes
>    printable = no
> ---- SNIP END: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf ----
> 
> My GNU/Linux client mounts the share with the following options:
> ---- SNIP START: /etc/fstab ----
> //virtue.local/files /media/network cifs \ 
> auto,user,pass=,rw,uid=1000,gid=100,noexec,nolock,\
> file_mode=0664,dir_mode=0775,iocharset=utf8,\
> wsize=57344,rsize=57344 0 0
> ---- SNIP END: /etc/fstab ----
> 
> The output of `smbd -d` is recorded here: http://pasted.at/fb1889588d_nl.html
> 
> All GNU/Linux clients use recent (2.6.32+) kernels and are x86_64 machines. 
> All clients are able to get the HTTP transfer speeds described above.
> 
> I'm still using the very same kernel that was used to build Samba/smbd on. 
> There's nothing out of the ordinary recorded in the server's logs.  `nmbd` 
> isn't running on the server, as I don't need that kind of name resolution 
> support. Samba is Version 3.4.5, running on Gentoo GNU/Linux ~amd64.
> 
> 
> If anyone spots something obvious that might limit transfer speeds in the way 
> I described, please leave a comment. Thanks very much in advance for your time 
> and effort!

Most likely, you're running into the lack of parallelism in Linux' CIFS
client. Writes are done in in turn currently and not in parallel as
they should be.

Fixing it is something we'd like to do, but it's a non-trivial amount
of work and it'll probably be a while before it's fixed.

-- 
Jeff Layton <jlayton at samba.org>