long startup times for large file systems

Tomasz Ciolek tmc at dreamcraft.com.au
Thu Dec 18 09:34:45 EST 2003


As far as I know rsync build a list of files and attributes + checksums
in memory. This can cause a large memory footprint while running the
process, so 72MB of data for 4.2 million files is only about right,
depending on the cheksum options, etc. I believe that average overhead
is about 120 bytes per file... someone correct me if I am wrong... 

Note that some options will consume more memory than other and some (like -c which asks rsync to chekcsum each file before sending ) will be very slow. As far as filesystems go Solaris 8 UFS is not the fastest of filesystems and traversing the directories may take some time expecially is there is other I/O happening on the FS.


On Wed, Dec 17, 2003 at 05:12:31PM -0500, Steve Howie wrote:
> Howdy,
> Rsync has been churning away for 45 mins, presumably bulding an in-core 
> list of files to be copied to the destination. This is a very very large 
> filesystem we are copying locally - approximately 4.2million files 
> (WebCT). The resident process size for rsync has grown to 72Mb. - is 
> this normal behaviour for a file system this size, and does rsync have 
> the ability to handle such a large number of files? The filesystem size 
> is relatively modest - 20gb. or so, but the millions of small files  
> might explain while its taking so long.
> I presume that once rsync has built it's in-memory list of files it 
> proceeds to copy the whole shebang over. This is the initial copy, 
> subsequent copies should be much faster.
> Also, are there any options suitable for turning off that would speed up 
> the whole process? We're using (as root)
>           # rsync -a --progress --stats   /global/webct/     /target
> Solaris 8 and rsync 2.5.7
> Thanks
> Scotty
> -- 
> To unsubscribe or change options: 
> http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Tomasz M. Ciolek	
  email:  tmc at dreamcraft dot com dot au 
	GPG Key ID: 0x41C4C2F0  Key available on www.pgp.net	
  Everything falls under the law of change;	
  Like a dream, a phantom, a bubble, a shadow,
  like dew or flash of lightning.
  You should contemplate like this. 

More information about the rsync mailing list