[clug] NFS slow cp/rm for many small files
Steve Walsh
Steve at nerdvana.org.au
Thu Oct 26 03:19:29 GMT 2006
Hi Steve
it looks to me like the bottleneck is either some part of the network or (long
shot) Kernel memory pages.
Check to see if you are getting UDP fragmentation, and if your Kernel is set for
16kb pages as opposed to 4kb.
Normally, the Linux NFS client uses read-ahead and delayed writes to hide the
latency of NFS read and write operations. However, the client can cache only a
single read or write request per page. So, if reading or writing a whole page
requires more than one on-the-wire read or write operation (which it certainly
does if rsize or wsize is 1024), each of these operations must complete before
the next one can be issued. In the case of small NFS Version 3 write operations,
the write must be a "FILE_SYNC" because the client must fully complete each
write before it issues the next one.
Has some part of your network auto-neg'd down to Half-Duplex (Collision City for
UDP!)? You can use eth-tool/mii-tool to check this out.
Noatime will reduce the need by the filesystem to update the atime information
when the file is read (file write = delay), but your delay appears to be in the
deleting and writing, so take a look at the above.
Steve
Steve Granger wrote:
> Hi
>
> I'm having problems copying/deleting large numbers of small files on an
> nfs partition in that it's taking an extremely long time to do so.
>
> E.g.
> steveg at steve-ubuntu:~/tmp$ du -sm linux-2.6.17-gentoo-r8
> 346 linux-2.6.17-gentoo-r8
>
> steveg at steve-ubuntu:~/tmp$ time cp -r linux-2.6.17-gentoo-r8 linux
>
> real 30m52.234s
> user 0m0.888s
> sys 0m14.809s
>
> steveg at steve-ubuntu:~/tmp$ time rm -rf linux
>
> real 9m36.636s
> user 0m0.292s
> sys 0m5.248s
>
>
> Gentoo server (192.168.10.6), Ubuntu clients (192.168.10.0/24).
>
> server linux # rpcinfo -p
> program vers proto port
> 100000 2 tcp 111 portmapper
> 100000 2 udp 111 portmapper
> 100024 1 udp 32765 status
> 100024 1 tcp 32765 status
> 100003 2 udp 2049 nfs
> 100003 3 udp 2049 nfs
> 100003 2 tcp 2049 nfs
> 100003 3 tcp 2049 nfs
> 100021 1 udp 4001 nlockmgr
> 100021 3 udp 4001 nlockmgr
> 100021 4 udp 4001 nlockmgr
> 100021 1 tcp 4001 nlockmgr
> 100021 3 tcp 4001 nlockmgr
> 100021 4 tcp 4001 nlockmgr
> 100005 1 udp 32767 mountd
> 100005 1 tcp 32767 mountd
> 100005 2 udp 32767 mountd
> 100005 2 tcp 32767 mountd
> 100005 3 udp 32767 mountd
> 100005 3 tcp 32767 mountd
>
> steveg at steve-ubuntu:~/tmp$ rpcinfo -p
> program vers proto port
> 100000 2 tcp 111 portmapper
> 100000 2 udp 111 portmapper
> 100021 1 udp 32768 nlockmgr
> 100021 3 udp 32768 nlockmgr
> 100021 4 udp 32768 nlockmgr
> 100021 1 tcp 43421 nlockmgr
> 100021 3 tcp 43421 nlockmgr
> 100021 4 tcp 43421 nlockmgr
> 100007 2 udp 731 ypbind
> 100007 1 udp 731 ypbind
> 100007 2 tcp 734 ypbind
> 100007 1 tcp 734 ypbind
> 100024 1 udp 859 status
> 100024 1 tcp 862 status
>
>
> The export on the server is
>
> /home 192.168.10.0/255.255.255.0(no_root_squash,rw,sync)
>
> clients mount it with this in /etc/fstab
>
> 192.168.10.6:/home /home nfs defaults,soft 0 1
>
> I am able to copy large files in a reasonable amount of time
>
> steveg at steve-ubuntu:~/tmp$ ls -la /media/sda1/tmp/cxslivecd-5.0.8.iso
> -rw-r--r-- 1 steveg users 514588672 2006-09-01 03:24
> /media/sda1/tmp/cxslivecd-5.0.8.iso
> steveg at steve-ubuntu:~/tmp$ time cp /media/sda1/tmp/cxslivecd-5.0.8.iso .
> real 1m15.187s
> user 0m0.024s
> sys 0m3.372s
> steveg at steve-ubuntu:~/tmp$
>
> This post suggests using the adjusting the rsize and wsize parameters
> and also adding the noatime to the mount statements.
> http://forum.sun.com/jive/thread.jspa?threadID=90909&messageID=314314
>
> Should I be adjusting these values? Another suggestion I saw was
> adjusting the MTU on the nfs server, could this be a complete was on
> time seeing that there are no errors reported on the server
>
> server linux # netstat -i
> Kernel Interface table
> Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
> TX-OVR Flg
> eth0 1500 0106259782 0 0 0154252353 0 0
> 0 BMRU
>
>
> The server /home partition is part of a RAID5 array... though I can't
> really consider the adaptec 4805SAS (aacraid) controller with the 15K
> Maxtor Atlas II drives attached to be a bottleneck.
>
> TIA
>
> --
> Steve
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
More information about the linux
mailing list