Question about rsync and BIG mirror

[Jamie Lokier]

Wayne Davison wrote:
> On Mon, Mar 06, 2006 at 07:18:45PM +0200, Shachar Shemesh wrote:
> > In fact, I know of at least one place where they don't use rsync because
> > they don't have enough RAM+SWAP to hold the list of files in memory.
> > 
> > As far as future directions for rsync, I think this is the major place
> > where rsync needs to become better.
> 
> I agree, and I am planning to add protocol improvements to rsync that
> will allow it to have a much smaller memory footprint, and to begin
> transferring files without pre-scanning the whole hierarchy first.

While you're there, one little trick I've found that speeds up
scanning large directory hierarchies is to stat() or open() entries in
inode-number order.  For some filesystems it makes no difference, but
for others it reduces the average disk seek time as on many common
filesystems, inode number is related to position on the disk.  In
unusual cases I've seen a factor of 10 improvement, but usually it's
just 1-2.

-- Jamie