long startup times for large file systems

jw schultz jw at pegasys.ws
Thu Dec 18 10:08:41 EST 2003

On Thu, Dec 18, 2003 at 09:34:45AM +1100, Tomasz Ciolek wrote:
> Scotty
> As far as I know rsync build a list of files and attributes + checksums
> in memory. This can cause a large memory footprint while running the
> process, so 72MB of data for 4.2 million files is only about right,
> depending on the cheksum options, etc. I believe that average overhead
> is about 120 bytes per file... someone correct me if I am wrong... 

Depending on filename lengths it is about 100 bytes per file.
100B * 4.2M = 420MB but his 72MB is only rss.

File checksums are only generated during file list build if
-c has been specified.  As you mention below.  The file checksum
would add 20 bytes per file to the footprint.

> Note that some options will consume more memory than other
> and some (like -c which asks rsync to chekcsum each file
> before sending ) will be very slow. As far as filesystems
> go Solaris 8 UFS is not the fastest of filesystems and
> traversing the directories may take some time expecially
> is there is other I/O happening on the FS.

More information about the rsync mailing list