File walking issue?

[Andrew Liles]

I assume
- the remote server is Win2K
- the remote server does not have an rsync daemon
- the remote server is accessed using Samba

The problem, I reckon, is that your entire set of files is being dragged 
across the network to the local machine during the "building file list" 
before it properly gets into copying files.

My simple understanding of what rsync does is this:
Frist: Rsync works out what to send by firstly determining which files 
exist on the source (this is a quick "ls" type operation).

Second: assume a file has an identical file size then rsync must examine 
the file to see if it is the same. It does, in essence, by doing a checksum 
on the file; I am not sure quite how rsync works, but let's assume it 
divides the file into blocks and does a checksum on the first block of 
bytes and compares that to the local copy's first block checksum.  How can 
it do that on the remote Win2K?  If you machine has no rsync daemon then 
your LOCAL machine needs to calculate the checksum for the REMOTE 
file.  The checksum is small to transmit over the network, but if the block 
is, say 100k, we incur a 100k(+a little bit) network transmission.  If the 
file is actually identical then WHOLE file has to be transmitted before it 
knows that that file does not need to be transferred!

So why use rsync if it is this "bad"?  Well much of the time (a) the remote 
runs an rsync daemon so the client can ask the remote to calculate the 
checksum, (b) the local can invoke on the remote a shell in which it can 
invoke rsync.  In the case of Win2K and a Samba share you don't have this 
possibility.

How should you solve it?  One way is to use "Cygwin" (www.cygwin.com) which 
allows you to run Unix commands (including rysnc) on a Windows platform.  I 
use rsync CLIENT on a Win2K box fine.  What I have not tried is running the 
daemon on the Win2K box.  Perhaps someone else can confirm this works ok.



At 18:43 16/02/2004, Max Kipness wrote:
>Hello,
>
>
>
>I'm having an issue with one particular server and am hoping someone
>here has dealt with this.
>
>
>
>I'm not sure whether this is a strictly samba issue or relates to the
>way rsync walks the file list.
>
>
>
>Basically after mounting a Windows 2000 file system using and then
>rsyncing the contents of this mount, it seems to take 5 - 8 hours to
>complete. I've checked on the log periodically and determined that it's
>the 'building of the file list' that is taking 95% of the time. We are
>only talking about 140,000 files. I do many samba shares and not of them
>have this issue. When doing a manual 'ls' command in various directories
>on the mount, I encounter no slowness or anything out of the ordinary.
>The samba log doesn't give much of a clue either.
>
>
>
>Has anybody come across this? Or does anybody have any ideas of how to
>troubleshoot?
>
>
>
>Oh, and I'm using Rsync 2.6
>
>
>
>Thanks,
>
>Max
>
>--
>To unsubscribe or change options: 
>http://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html