Rsync: Re: patch to enable faster mirroring of large filesystems

[Dave Dykstra]

On Tue, Nov 27, 2001 at 05:00:14PM -0500, Lenny Foner wrote:
...
>     [ . . . ]
> 
>     I'm pretty sure that rsync won't use up memory for excluded files so it
>     would make no difference.
> 
> ...though this also implies (since you say it'd probably use basically
> the same mechanism internally) that it -would- nonetheless keep info
> around for the entire run about each file that -was- going to be/had been
> transferred, yes?

Yes, that is a limitation in the current implementation.  Long ago the
original author Andrew Tridgell proposed a rewrite to avoid that but he
never got to it.


> This is a separate problem from how the files are
> selected, but I've lost track of what the right solution here should
> be, except for dropping each directory's info after you leave it---
> which would presumably not necessarily be easy if you're getting the
> file list in arbitrary order via --files-from, but might be easier
> if they were being generated via rsync's current traversal algorithm.

I'm not clear on the details of why rsync keeps some information in memory
for every file, but I don't think it has anything to do with order or
per-directory limitations.


> In any event, I -hope- that the memory issue is cleanly separable
> from the issue of how files get selected; this might be a good time
> to at least ponder the issue, if --files-from might soon exist.

I believe the two are cleanly separable.

- Dave