[clug] NFS slow cp/rm for many small files

Steve Walsh Steve at nerdvana.org.au
Thu Oct 26 03:19:29 GMT 2006


Hi Steve

it looks to me like the bottleneck is either some part of the network or (long 
shot) Kernel memory pages.

Check to see if you are getting UDP fragmentation, and if your Kernel is set for 
16kb pages as opposed to 4kb.

Normally, the Linux NFS client uses read-ahead and delayed writes to hide the 
latency of NFS read and write operations. However, the client can cache only a 
single read or write request per page. So, if reading or writing a whole page 
requires more than one on-the-wire read or write operation (which it certainly 
does if rsize or wsize is 1024), each of these operations must complete before 
the next one can be issued. In the case of small NFS Version 3 write operations, 
the write must be a "FILE_SYNC" because the client must fully complete each 
write before it issues the next one.

Has some part of your network auto-neg'd down to Half-Duplex (Collision City for 
UDP!)? You can use eth-tool/mii-tool to check this out.

Noatime will reduce the need by the filesystem to update the atime information 
when the file is read (file write = delay), but your delay appears to be in the 
deleting and writing, so take a look at the above.

Steve

Steve Granger wrote:
> Hi
> 
> I'm having problems copying/deleting large numbers of small files on an
> nfs partition in that it's taking an extremely long time to do so.
> 
> E.g.
> steveg at steve-ubuntu:~/tmp$ du -sm  linux-2.6.17-gentoo-r8
> 346     linux-2.6.17-gentoo-r8
> 
> steveg at steve-ubuntu:~/tmp$ time cp -r linux-2.6.17-gentoo-r8 linux
> 
> real    30m52.234s
> user    0m0.888s
> sys     0m14.809s
> 
> steveg at steve-ubuntu:~/tmp$ time rm -rf  linux
> 
> real    9m36.636s
> user    0m0.292s
> sys     0m5.248s
> 
> 
> Gentoo server (192.168.10.6), Ubuntu clients (192.168.10.0/24).
> 
> server linux # rpcinfo -p
>    program vers proto   port
>     100000    2   tcp    111  portmapper
>     100000    2   udp    111  portmapper
>     100024    1   udp  32765  status
>     100024    1   tcp  32765  status
>     100003    2   udp   2049  nfs
>     100003    3   udp   2049  nfs
>     100003    2   tcp   2049  nfs
>     100003    3   tcp   2049  nfs
>     100021    1   udp   4001  nlockmgr
>     100021    3   udp   4001  nlockmgr
>     100021    4   udp   4001  nlockmgr
>     100021    1   tcp   4001  nlockmgr
>     100021    3   tcp   4001  nlockmgr
>     100021    4   tcp   4001  nlockmgr
>     100005    1   udp  32767  mountd
>     100005    1   tcp  32767  mountd
>     100005    2   udp  32767  mountd
>     100005    2   tcp  32767  mountd
>     100005    3   udp  32767  mountd
>     100005    3   tcp  32767  mountd
> 
> steveg at steve-ubuntu:~/tmp$ rpcinfo -p
>    program vers proto   port
>     100000    2   tcp    111  portmapper
>     100000    2   udp    111  portmapper
>     100021    1   udp  32768  nlockmgr
>     100021    3   udp  32768  nlockmgr
>     100021    4   udp  32768  nlockmgr
>     100021    1   tcp  43421  nlockmgr
>     100021    3   tcp  43421  nlockmgr
>     100021    4   tcp  43421  nlockmgr
>     100007    2   udp    731  ypbind
>     100007    1   udp    731  ypbind
>     100007    2   tcp    734  ypbind
>     100007    1   tcp    734  ypbind
>     100024    1   udp    859  status
>     100024    1   tcp    862  status
> 
> 
> The export on the server is
> 
> /home   192.168.10.0/255.255.255.0(no_root_squash,rw,sync)
> 
> clients mount it with this in /etc/fstab
> 
> 192.168.10.6:/home /home        nfs     defaults,soft   0       1
> 
> I am able to copy large files in a reasonable amount of time
> 
> steveg at steve-ubuntu:~/tmp$ ls -la  /media/sda1/tmp/cxslivecd-5.0.8.iso
> -rw-r--r-- 1 steveg users 514588672 2006-09-01 03:24
> /media/sda1/tmp/cxslivecd-5.0.8.iso
> steveg at steve-ubuntu:~/tmp$ time cp   /media/sda1/tmp/cxslivecd-5.0.8.iso .
> real    1m15.187s
> user    0m0.024s
> sys     0m3.372s
> steveg at steve-ubuntu:~/tmp$
> 
> This post suggests using the adjusting the rsize and wsize parameters
> and also adding the noatime to the mount statements.
> http://forum.sun.com/jive/thread.jspa?threadID=90909&messageID=314314
> 
> Should I be adjusting these values? Another suggestion I saw was
> adjusting the MTU on the nfs server, could this be a complete was on
> time seeing that there are no errors reported on the server
> 
> server linux # netstat -i
> Kernel Interface table
> Iface     MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP
> TX-OVR Flg
> eth0       1500   0106259782      0      0      0154252353      0      0
>      0 BMRU
> 
> 
> The server /home partition is part of a RAID5 array... though I can't
> really consider the adaptec 4805SAS (aacraid) controller with the 15K
> Maxtor Atlas II drives attached to be a bottleneck.
> 
> TIA
> 
> --
> Steve
> 
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email 
> ______________________________________________________________________


More information about the linux mailing list