[clug] NFS slow cp/rm for many small files

[Steve Granger]

Hi

I'm having problems copying/deleting large numbers of small files on an
nfs partition in that it's taking an extremely long time to do so.

E.g.
steveg at steve-ubuntu:~/tmp$ du -sm  linux-2.6.17-gentoo-r8
346     linux-2.6.17-gentoo-r8

steveg at steve-ubuntu:~/tmp$ time cp -r linux-2.6.17-gentoo-r8 linux

real    30m52.234s
user    0m0.888s
sys     0m14.809s

steveg at steve-ubuntu:~/tmp$ time rm -rf  linux

real    9m36.636s
user    0m0.292s
sys     0m5.248s


Gentoo server (192.168.10.6), Ubuntu clients (192.168.10.0/24).

server linux # rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  32765  status
    100024    1   tcp  32765  status
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100021    1   udp   4001  nlockmgr
    100021    3   udp   4001  nlockmgr
    100021    4   udp   4001  nlockmgr
    100021    1   tcp   4001  nlockmgr
    100021    3   tcp   4001  nlockmgr
    100021    4   tcp   4001  nlockmgr
    100005    1   udp  32767  mountd
    100005    1   tcp  32767  mountd
    100005    2   udp  32767  mountd
    100005    2   tcp  32767  mountd
    100005    3   udp  32767  mountd
    100005    3   tcp  32767  mountd

steveg at steve-ubuntu:~/tmp$ rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100021    1   udp  32768  nlockmgr
    100021    3   udp  32768  nlockmgr
    100021    4   udp  32768  nlockmgr
    100021    1   tcp  43421  nlockmgr
    100021    3   tcp  43421  nlockmgr
    100021    4   tcp  43421  nlockmgr
    100007    2   udp    731  ypbind
    100007    1   udp    731  ypbind
    100007    2   tcp    734  ypbind
    100007    1   tcp    734  ypbind
    100024    1   udp    859  status
    100024    1   tcp    862  status


The export on the server is

/home   192.168.10.0/255.255.255.0(no_root_squash,rw,sync)

clients mount it with this in /etc/fstab

192.168.10.6:/home /home        nfs     defaults,soft   0       1

I am able to copy large files in a reasonable amount of time

steveg at steve-ubuntu:~/tmp$ ls -la  /media/sda1/tmp/cxslivecd-5.0.8.iso
-rw-r--r-- 1 steveg users 514588672 2006-09-01 03:24
/media/sda1/tmp/cxslivecd-5.0.8.iso
steveg at steve-ubuntu:~/tmp$ time cp   /media/sda1/tmp/cxslivecd-5.0.8.iso .
real    1m15.187s
user    0m0.024s
sys     0m3.372s
steveg at steve-ubuntu:~/tmp$

This post suggests using the adjusting the rsize and wsize parameters
and also adding the noatime to the mount statements.
http://forum.sun.com/jive/thread.jspa?threadID=90909&messageID=314314

Should I be adjusting these values? Another suggestion I saw was
adjusting the MTU on the nfs server, could this be a complete was on
time seeing that there are no errors reported on the server

server linux # netstat -i
Kernel Interface table
Iface     MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP
TX-OVR Flg
eth0       1500   0106259782      0      0      0154252353      0      0
     0 BMRU


The server /home partition is part of a RAID5 array... though I can't
really consider the adaptec 4805SAS (aacraid) controller with the 15K
Maxtor Atlas II drives attached to be a bottleneck.

TIA

--
Steve

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________