[clug] NFS slow cp/rm for many small files

[Robert Edwards]

Steve,

If you want the NFS to go faster, turn the "sync" in the export options
to "async". Slightly less robust in the event of a file-server crash,
but much faster write throughput.

We run all of our (writable) NFS servers with "async" and have done for
years with never a problem (and some of them are _very_ busy).

Cheers,

Bob Edwards.

Steve Granger wrote:
> Hi
> 
> I'm having problems copying/deleting large numbers of small files on an
> nfs partition in that it's taking an extremely long time to do so.
> 
> E.g.
> steveg at steve-ubuntu:~/tmp$ du -sm  linux-2.6.17-gentoo-r8
> 346     linux-2.6.17-gentoo-r8
> 
> steveg at steve-ubuntu:~/tmp$ time cp -r linux-2.6.17-gentoo-r8 linux
> 
> real    30m52.234s
> user    0m0.888s
> sys     0m14.809s
> 
> steveg at steve-ubuntu:~/tmp$ time rm -rf  linux
> 
> real    9m36.636s
> user    0m0.292s
> sys     0m5.248s
> 
> 
> Gentoo server (192.168.10.6), Ubuntu clients (192.168.10.0/24).
> 
> server linux # rpcinfo -p
>    program vers proto   port
>     100000    2   tcp    111  portmapper
>     100000    2   udp    111  portmapper
>     100024    1   udp  32765  status
>     100024    1   tcp  32765  status
>     100003    2   udp   2049  nfs
>     100003    3   udp   2049  nfs
>     100003    2   tcp   2049  nfs
>     100003    3   tcp   2049  nfs
>     100021    1   udp   4001  nlockmgr
>     100021    3   udp   4001  nlockmgr
>     100021    4   udp   4001  nlockmgr
>     100021    1   tcp   4001  nlockmgr
>     100021    3   tcp   4001  nlockmgr
>     100021    4   tcp   4001  nlockmgr
>     100005    1   udp  32767  mountd
>     100005    1   tcp  32767  mountd
>     100005    2   udp  32767  mountd
>     100005    2   tcp  32767  mountd
>     100005    3   udp  32767  mountd
>     100005    3   tcp  32767  mountd
> 
> steveg at steve-ubuntu:~/tmp$ rpcinfo -p
>    program vers proto   port
>     100000    2   tcp    111  portmapper
>     100000    2   udp    111  portmapper
>     100021    1   udp  32768  nlockmgr
>     100021    3   udp  32768  nlockmgr
>     100021    4   udp  32768  nlockmgr
>     100021    1   tcp  43421  nlockmgr
>     100021    3   tcp  43421  nlockmgr
>     100021    4   tcp  43421  nlockmgr
>     100007    2   udp    731  ypbind
>     100007    1   udp    731  ypbind
>     100007    2   tcp    734  ypbind
>     100007    1   tcp    734  ypbind
>     100024    1   udp    859  status
>     100024    1   tcp    862  status
> 
> 
> The export on the server is
> 
> /home   192.168.10.0/255.255.255.0(no_root_squash,rw,sync)
> 
> clients mount it with this in /etc/fstab
> 
> 192.168.10.6:/home /home        nfs     defaults,soft   0       1
> 
> I am able to copy large files in a reasonable amount of time
> 
> steveg at steve-ubuntu:~/tmp$ ls -la  /media/sda1/tmp/cxslivecd-5.0.8.iso
> -rw-r--r-- 1 steveg users 514588672 2006-09-01 03:24
> /media/sda1/tmp/cxslivecd-5.0.8.iso
> steveg at steve-ubuntu:~/tmp$ time cp   /media/sda1/tmp/cxslivecd-5.0.8.iso .
> real    1m15.187s
> user    0m0.024s
> sys     0m3.372s
> steveg at steve-ubuntu:~/tmp$
> 
> This post suggests using the adjusting the rsize and wsize parameters
> and also adding the noatime to the mount statements.
> http://forum.sun.com/jive/thread.jspa?threadID=90909&messageID=314314
> 
> Should I be adjusting these values? Another suggestion I saw was
> adjusting the MTU on the nfs server, could this be a complete was on
> time seeing that there are no errors reported on the server
> 
> server linux # netstat -i
> Kernel Interface table
> Iface     MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP
> TX-OVR Flg
> eth0       1500   0106259782      0      0      0154252353      0      0
>      0 BMRU
> 
> 
> The server /home partition is part of a RAID5 array... though I can't
> really consider the adaptec 4805SAS (aacraid) controller with the 15K
> Maxtor Atlas II drives attached to be a bottleneck.
> 
> TIA
> 
> --
> Steve
> 
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email 
> ______________________________________________________________________