TODO hardlink performance optimizations
jw at pegasys.ws
Mon Jan 5 00:07:10 GMT 2004
On Sun, Jan 04, 2004 at 06:35:03AM -0600, John Van Essen wrote:
> You articulated your situation clear enough for me. Thanks.
> I'll address your issue about when rsync is running locally for /vol/N
> to /vol/N_mirror syncing, it exhausts all of the RAM and swap.
> If you haven't read jw schultz's "How Rsync Works" page, here the link:
> The sender, receiver, and generator each have a full copy of the file
> list (each file's entry uses 100 bytes on average).
The reciever and generator start out shareing the file list
but dirverge as the receiver modifies directory entries
forcing copy-on-write. This COW doesn't just copy the
file_struct but also the strings because they are
interspersed. Wayne has a patch in CVS to prevent
> Additonally, the --hard-links option creates yet *another* full copy of
> the file list in the receiver, so that's even more memory consumed.
That is a partial copy. Only the actual file_struct is
copied at 72bytes (32bit linux) per file, the strings are
not copied. My patch (and John's i assume) reduces that to
just copying the file_list index which is 4bytes (32bit) per
J.W. Schultz Pegasystems Technologies
email address: jw at pegasys.ws
Remember Cernan and Schmitt
More information about the rsync