long startup times for large file systems
jw schultz
jw at pegasys.ws
Thu Dec 18 10:08:41 EST 2003
On Thu, Dec 18, 2003 at 09:34:45AM +1100, Tomasz Ciolek wrote:
> Scotty
>
> As far as I know rsync build a list of files and attributes + checksums
> in memory. This can cause a large memory footprint while running the
> process, so 72MB of data for 4.2 million files is only about right,
> depending on the cheksum options, etc. I believe that average overhead
> is about 120 bytes per file... someone correct me if I am wrong...
Depending on filename lengths it is about 100 bytes per file.
100B * 4.2M = 420MB but his 72MB is only rss.
File checksums are only generated during file list build if
-c has been specified. As you mention below. The file checksum
would add 20 bytes per file to the footprint.
> Note that some options will consume more memory than other
> and some (like -c which asks rsync to chekcsum each file
> before sending ) will be very slow. As far as filesystems
> go Solaris 8 UFS is not the fastest of filesystems and
> traversing the directories may take some time expecially
> is there is other I/O happening on the FS.
More information about the rsync
mailing list