[linux-cifs-client] CIFS slow on gigabit, doesn't support sockopt=TCP_NODELAY ?

Jeff Layton jlayton at redhat.com
Wed Aug 26 04:06:23 MDT 2009


On Tue, 25 Aug 2009 15:45:19 -0400
Timothy Normand Miller <theosib at gmail.com> wrote:

> Here's my "patch" to fs/cifs/connect.c:
> 
>         /*
>          * Eventually check for other socket options to change from
>          *  the default. sock_setsockopt not used because it expects
>          *  user space buffer
>          */
>         socket->sk->sk_rcvtimeo = 7 * HZ;
>         socket->sk->sk_sndtimeo = 5 * HZ;
> 
> +        rc = kernel_setsockopt(socket, SOL_TCP, TCP_NODELAY, (char
> __user *)&val, sizeof(val));
> +        printk(KERN_WARNING "CIFS: TCP_NODELAY returns %d\n", rc);
> +
> +        val = IPTOS_LOWDELAY;
> +        rc = kernel_setsockopt(socket, SOL_IP, IP_TOS, (char __user
> *)&val, sizeof(val));
> +        printk(KERN_WARNING "CIFS: IPTOS_LOWDELAY returns %d\n", rc);
> 
> 
> Both calls return zero, possibly suggesting that they succeeded.  When
> NODELAY alone, I didn't notice any improvement.  With both, I notice
> roughly an 11% speedup, with improved _consistency_ in throughput as
> measured by iozone.  This is still about 1/2 the speed I can get
> accessing the same share from my Mac.
> 

What exactly are you measuring here? Reads? Writes?

What about with just IPTOS_LOWDELAY alone? To answer your earlier
question, see the TOS mangle target in the iptables manpage, or just
google around for "iptables TOS bits".

> There are several possibilities that come to mind:
> 
> (1) [most probable] I'm doing it wrong.

No, looks like you're setting the socket options correctly.

> (2) There's a bug or design defficiency in CIFS that's causing this
> (3) The bottleneck(s) in CIFS are somewhere other than the networking

That's what I'd suspect. Lack of parallelism in reads/writes is
probably the main bottleneck here, and not network throughput. Nagle's
algorithm in particular shouldn't be an issue at all if you're
keeping the socket stuffed with enough outgoing data.

> (4) The SMB protocol itself is deficient, and the Mac is using some
> other protocol to access the share
> 

I doubt this is the case, but you should probably get familiar with
wireshark if you're going to poke around in this area.

> 
> The reason I suspected that TCP_NODELAY and IPTOS_LOWDELAY were
> involved was that because I enabled those options on the Linux box's
> SMB server, 10 megs/sec was the limit I could get between the Mac and
> the Linux box over SMB.  When enabling those options, I then became
> limited by the disks (typically about 50 megs/sec for writes, faster
> for reads from the RAID1).  With the AEBS share, the speed was also
> limited to about 10 megs/sec, which I didn't think was a coincidence.
> Since the Mac can move more like 20 megs/sec to/from the AEBS (limited
> by the fact that these are USB drives), my suspicion is that I simply
> haven't done it right.
> 
> How can I be sure I'm setting these options correctly and on the right
> socket and that it applies to both directions, etc?



-- 
Jeff Layton <jlayton at redhat.com>


More information about the linux-cifs-client mailing list