very big rsync only worked partially what are size limitations?

mlaks at verizon.net mlaks at verizon.net
Sun Dec 19 17:32:34 GMT 2004


Dear Wayne,
On Sunday 19 December 2004 11:18 am, you wrote:
> On Sat, Dec 18, 2004 at 09:50:53PM -0500, mlaks at verizon.net wrote:
> > I then tried again and started the rsync script on the directory
> > again, and it ended without copying over any more stuff, as far as I
> > can see  - size wise and number of subdirectories.  :(.
>
> Did it output an error? Or end normally? If it did not output an error,
> it believes that the files you asked it to transfer are up-to-date.
Since it takes a long time, and i am logging in via dialup, I don't see how to 
capture errors, particularly those at the other end... I am reading a nice 
talk by Dr. A Tridgell in which he says to use the --rsync-path command to 
substitute a wrapper script on the destination in order to do cool things 
before and after running the rsync, but i have not figured out how to do this 
so that i can capture the error messages on destination machine... I know 
that there are messages, cause i was logged on once onto the destination 
while i was doing a rsync and i saw errors due to permission problems on the 
destination directory. So if you have an easy sample script to use to capture 
them i will be very grateful! 

I also notice that my machine is running redhat 7.3 with rsync 2.5.4. I am 
interested in moving my machines over to debian sarge, (as soon as I get the 
machines backed up :) ). Then i will have a  more modern rsync!

So far, i reran rsync using a perl script and the subdirectory approach 
(infinitely less memory usage) and in fact the sizes of the total directories 
seem to be roughly the same (i wanted to get the 2 df values on the two 
machines to be equal, but at least they are very close).

By the way that raises a second question: 
How can i verify that i have rsync'd successfully with a second method other 
than just rsync again. 
I note that now rsync ends very fast, meaning that rsync  is certain all is 
ok. 
Is there a well known better second check i can do, other than look for 
identical du or df?
Thanks !!!
Mitchell
Thanks
>
> > So my question, what are the size limitations on this sort of stupid
> > original way of doing it?
>
> The maximum file count is mainly limited by memory.  Figure that it
> probably takes 100 bytes or so per file/directory in the transfer list.
> If you're not using a modern rsync (2.6.3 is the latest) this memory
> requirement will be much higher, particularly on the receiving system:
> the receiving side forks off an extra rsync process, which should share
> a lot of the same memory as the parent process (if your fork uses copy-
> on-write memory-sharing).  Depending on your OS and the age of your
> rsync (modern ones do a better job of keeping the memory shared between
> the processes), the receiving side's memory use could bloat to be double
> what the sending side needs.  Your best bet is to just run the command
> and look at memory use, keeping an eye on the total proces size and (on
> the receiving side) how much memory is shared.
>
> ..wayne..


More information about the rsync mailing list