very big rsync only worked partially what are size limitations?
mlaks at verizon.net
mlaks at verizon.net
Sun Dec 19 17:32:34 GMT 2004
On Sunday 19 December 2004 11:18 am, you wrote:
> On Sat, Dec 18, 2004 at 09:50:53PM -0500, mlaks at verizon.net wrote:
> > I then tried again and started the rsync script on the directory
> > again, and it ended without copying over any more stuff, as far as I
> > can see - size wise and number of subdirectories. :(.
> Did it output an error? Or end normally? If it did not output an error,
> it believes that the files you asked it to transfer are up-to-date.
Since it takes a long time, and i am logging in via dialup, I don't see how to
capture errors, particularly those at the other end... I am reading a nice
talk by Dr. A Tridgell in which he says to use the --rsync-path command to
substitute a wrapper script on the destination in order to do cool things
before and after running the rsync, but i have not figured out how to do this
so that i can capture the error messages on destination machine... I know
that there are messages, cause i was logged on once onto the destination
while i was doing a rsync and i saw errors due to permission problems on the
destination directory. So if you have an easy sample script to use to capture
them i will be very grateful!
I also notice that my machine is running redhat 7.3 with rsync 2.5.4. I am
interested in moving my machines over to debian sarge, (as soon as I get the
machines backed up :) ). Then i will have a more modern rsync!
So far, i reran rsync using a perl script and the subdirectory approach
(infinitely less memory usage) and in fact the sizes of the total directories
seem to be roughly the same (i wanted to get the 2 df values on the two
machines to be equal, but at least they are very close).
By the way that raises a second question:
How can i verify that i have rsync'd successfully with a second method other
than just rsync again.
I note that now rsync ends very fast, meaning that rsync is certain all is
Is there a well known better second check i can do, other than look for
identical du or df?
> > So my question, what are the size limitations on this sort of stupid
> > original way of doing it?
> The maximum file count is mainly limited by memory. Figure that it
> probably takes 100 bytes or so per file/directory in the transfer list.
> If you're not using a modern rsync (2.6.3 is the latest) this memory
> requirement will be much higher, particularly on the receiving system:
> the receiving side forks off an extra rsync process, which should share
> a lot of the same memory as the parent process (if your fork uses copy-
> on-write memory-sharing). Depending on your OS and the age of your
> rsync (modern ones do a better job of keeping the memory shared between
> the processes), the receiving side's memory use could bloat to be double
> what the sending side needs. Your best bet is to just run the command
> and look at memory use, keeping an eye on the total proces size and (on
> the receiving side) how much memory is shared.
More information about the rsync