long startup times for large file systems
Tomasz Ciolek
tmc at dreamcraft.com.au
Thu Dec 18 09:34:45 EST 2003
Scotty
As far as I know rsync build a list of files and attributes + checksums
in memory. This can cause a large memory footprint while running the
process, so 72MB of data for 4.2 million files is only about right,
depending on the cheksum options, etc. I believe that average overhead
is about 120 bytes per file... someone correct me if I am wrong...
Note that some options will consume more memory than other and some (like -c which asks rsync to chekcsum each file before sending ) will be very slow. As far as filesystems go Solaris 8 UFS is not the fastest of filesystems and traversing the directories may take some time expecially is there is other I/O happening on the FS.
Tomasz
On Wed, Dec 17, 2003 at 05:12:31PM -0500, Steve Howie wrote:
> Howdy,
>
> Rsync has been churning away for 45 mins, presumably bulding an in-core
> list of files to be copied to the destination. This is a very very large
> filesystem we are copying locally - approximately 4.2million files
> (WebCT). The resident process size for rsync has grown to 72Mb. - is
> this normal behaviour for a file system this size, and does rsync have
> the ability to handle such a large number of files? The filesystem size
> is relatively modest - 20gb. or so, but the millions of small files
> might explain while its taking so long.
>
> I presume that once rsync has built it's in-memory list of files it
> proceeds to copy the whole shebang over. This is the initial copy,
> subsequent copies should be much faster.
>
> Also, are there any options suitable for turning off that would speed up
> the whole process? We're using (as root)
>
> # rsync -a --progress --stats /global/webct/ /target
>
> Solaris 8 and rsync 2.5.7
>
> Thanks
>
> Scotty
>
> --
> To unsubscribe or change options:
> http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Tomasz M. Ciolek
*******************************************************************************
email: tmc at dreamcraft dot com dot au
*******************************************************************************
GPG Key ID: 0x41C4C2F0 Key available on www.pgp.net
*******************************************************************************
Everything falls under the law of change;
Like a dream, a phantom, a bubble, a shadow,
like dew or flash of lightning.
You should contemplate like this.
More information about the rsync
mailing list