long startup times for large file systems
jw at pegasys.ws
Thu Dec 18 09:32:50 EST 2003
On Wed, Dec 17, 2003 at 05:12:31PM -0500, Steve Howie wrote:
> Rsync has been churning away for 45 mins, presumably bulding an in-core
> list of files to be copied to the destination. This is a very very large
> filesystem we are copying locally - approximately 4.2million files
> (WebCT). The resident process size for rsync has grown to 72Mb. - is
> this normal behaviour for a file system this size, and does rsync have
> the ability to handle such a large number of files? The filesystem size
> is relatively modest - 20gb. or so, but the millions of small files
> might explain while its taking so long.
You have sussed it. Rsync is building a complete in-memory
file list with all metadata. This takes approximately 100
bytes per path (directories too) 4.2 million files is going
to take over 400MB. You might want to break the job up.
> I presume that once rsync has built it's in-memory list of files it
> proceeds to copy the whole shebang over. This is the initial copy,
> subsequent copies should be much faster.
Building the file list takes the same amount of time every
time. The speedup in subsequent syncs will be in the
selective data transfers.
> Also, are there any options suitable for turning off that would speed up
> the whole process? We're using (as root)
> # rsync -a --progress --stats /global/webct/ /target
Is /global/webct by any chance served by NFS? That could
account for it taking so much time just walking the file
list. Whenever possible it is best to avoid rsyncing over
> Solaris 8 and rsync 2.5.7
To be honest i'd use cp -a or cpio for at least the initial
transfer. Rsync does work for local copy but is actually
slower than other methods. It is over the network (the R in
rsync) that it shines.
J.W. Schultz Pegasystems Technologies
email address: jw at pegasys.ws
Remember Cernan and Schmitt
More information about the rsync