proposal to speed rsync with lots of files

[Peter Salameh]

Kyle Lanclos wrote:
> Peter Salameh wrote:
>   
>> One of the speed-limiting issues with rsync is having to send huge file 
>> lists when mirroring large file systems, even for incremental updates 
>> where only a small part of the file system might have changed.
>>     
>
> Personally, I find that the sending of the file list, whether incremental
> or otherwise, takes orders of magnitude less time than the construction of
> the file list in the first place. The act of stat'ing millions of files
> takes an enormous amount of time in comparison to just about anything else,
> assuming that you are not on a low-bandwidth link.
>   
Many of the locations I mirror do have low-bandwidth links, and I 
imagine that a sizable fraction of rsync users have issues with 
bandwidth at least some of the time.  I have not studied rsync internals 
either, but I imagine using a checksum to avoid sending directory file 
lists unnecessarily would not hurt anything and would help folks with 
lower bandwidth.

Peter

-------------- next part --------------
HTML attachment scrubbed and removed