File walking issue?
andrew.liles at starfishzone.com
Mon Feb 16 23:19:25 GMT 2004
- the remote server is Win2K
- the remote server does not have an rsync daemon
- the remote server is accessed using Samba
The problem, I reckon, is that your entire set of files is being dragged
across the network to the local machine during the "building file list"
before it properly gets into copying files.
My simple understanding of what rsync does is this:
Frist: Rsync works out what to send by firstly determining which files
exist on the source (this is a quick "ls" type operation).
Second: assume a file has an identical file size then rsync must examine
the file to see if it is the same. It does, in essence, by doing a checksum
on the file; I am not sure quite how rsync works, but let's assume it
divides the file into blocks and does a checksum on the first block of
bytes and compares that to the local copy's first block checksum. How can
it do that on the remote Win2K? If you machine has no rsync daemon then
your LOCAL machine needs to calculate the checksum for the REMOTE
file. The checksum is small to transmit over the network, but if the block
is, say 100k, we incur a 100k(+a little bit) network transmission. If the
file is actually identical then WHOLE file has to be transmitted before it
knows that that file does not need to be transferred!
So why use rsync if it is this "bad"? Well much of the time (a) the remote
runs an rsync daemon so the client can ask the remote to calculate the
checksum, (b) the local can invoke on the remote a shell in which it can
invoke rsync. In the case of Win2K and a Samba share you don't have this
How should you solve it? One way is to use "Cygwin" (www.cygwin.com) which
allows you to run Unix commands (including rysnc) on a Windows platform. I
use rsync CLIENT on a Win2K box fine. What I have not tried is running the
daemon on the Win2K box. Perhaps someone else can confirm this works ok.
At 18:43 16/02/2004, Max Kipness wrote:
>I'm having an issue with one particular server and am hoping someone
>here has dealt with this.
>I'm not sure whether this is a strictly samba issue or relates to the
>way rsync walks the file list.
>Basically after mounting a Windows 2000 file system using and then
>rsyncing the contents of this mount, it seems to take 5 - 8 hours to
>complete. I've checked on the log periodically and determined that it's
>the 'building of the file list' that is taking 95% of the time. We are
>only talking about 140,000 files. I do many samba shares and not of them
>have this issue. When doing a manual 'ls' command in various directories
>on the mount, I encounter no slowness or anything out of the ordinary.
>The samba log doesn't give much of a clue either.
>Has anybody come across this? Or does anybody have any ideas of how to
>Oh, and I'm using Rsync 2.6
>To unsubscribe or change options:
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
More information about the rsync