Rsync: Re: patch to enable faster mirroring of large filesystems
dwd at bell-labs.com
Thu Nov 29 05:29:34 EST 2001
On Tue, Nov 27, 2001 at 05:00:14PM -0500, Lenny Foner wrote:
> [ . . . ]
> I'm pretty sure that rsync won't use up memory for excluded files so it
> would make no difference.
> ...though this also implies (since you say it'd probably use basically
> the same mechanism internally) that it -would- nonetheless keep info
> around for the entire run about each file that -was- going to be/had been
> transferred, yes?
Yes, that is a limitation in the current implementation. Long ago the
original author Andrew Tridgell proposed a rewrite to avoid that but he
never got to it.
> This is a separate problem from how the files are
> selected, but I've lost track of what the right solution here should
> be, except for dropping each directory's info after you leave it---
> which would presumably not necessarily be easy if you're getting the
> file list in arbitrary order via --files-from, but might be easier
> if they were being generated via rsync's current traversal algorithm.
I'm not clear on the details of why rsync keeps some information in memory
for every file, but I don't think it has anything to do with order or
> In any event, I -hope- that the memory issue is cleanly separable
> from the issue of how files get selected; this might be a good time
> to at least ponder the issue, if --files-from might soon exist.
I believe the two are cleanly separable.
More information about the rsync