proposal to speed rsync with lots of files
Peter Salameh
psalameh at ucsd.edu
Fri Mar 6 00:40:39 GMT 2009
Kyle Lanclos wrote:
> Peter Salameh wrote:
>
>> One of the speed-limiting issues with rsync is having to send huge file
>> lists when mirroring large file systems, even for incremental updates
>> where only a small part of the file system might have changed.
>>
>
> Personally, I find that the sending of the file list, whether incremental
> or otherwise, takes orders of magnitude less time than the construction of
> the file list in the first place. The act of stat'ing millions of files
> takes an enormous amount of time in comparison to just about anything else,
> assuming that you are not on a low-bandwidth link.
>
Many of the locations I mirror do have low-bandwidth links, and I
imagine that a sizable fraction of rsync users have issues with
bandwidth at least some of the time. I have not studied rsync internals
either, but I imagine using a checksum to avoid sending directory file
lists unnecessarily would not hurt anything and would help folks with
lower bandwidth.
Peter
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the rsync
mailing list