Local disk rsync

[listserv.traffic at sloop.net]

Hello Tony,

Friday, April 10, 2009, 11:39:07 AM, you wrote:

> listserv.traffic at sloop.net wrote:
>> 
>> I've done quite a bit of looking, but I haven't found an answer that
>> answers this question.
>> 
>> Environment:
>> cygwin on Windows
>> rsync 3.0.4
>> 
>> I know that rsync isn't optimzed for speed on local copies - that's
>> clear in my testing. I'm attemting to sync a large volume of files.
>> 
>> (In this case, I'm syncing a rdiff-backup set...)
>> 
>> An initial sync will be about one fourth as fast a a cp of the same
>> files. "Ok, that's not a huge deal. Subsequent syncs should be a lot
>> faster" I though.
>> 
>> No.
>> 
>> Using a native windows sync tool, a sync of the source to the
>> destination takes perhaps 5 minutes. (~24G, 80K files) However using
>> rsync takes a very long time. (I've never let it finish, but it's
>> still running an hour later or more...)
>> 
>> Here's how I run it:
>> rsync --recursive --delete --verbose --stats --perms --super
>> /cygdrive/e/somedir/ /cygdrive/f/somedir/
>> 
>> Thoughts?
> Any copying, local, from and to the same spindle looks like;

> read some stuff
> seek to the other place
> write some stuff
> seek to the other place

> Timing should be dominated by how much stuff in each place before it seeks
> to the new place.
> For Windows, ROBOCOPY may do what you want  (Resource kit XP and below,
> native on Vista)
> You will probably want the switches /R:1 and /W:1 or similare.

>> 
>> I'm mainly using rsync 'cause I know it, and use it. If there's some
>> alternative I'd be glad for a pointer. I don't know of any other good
>> sync tools like rsync. (Unison I guess...though I only need one way
>> sync.) [Whole file copy is fine if needed, though some of the files
>> are large 1 GB+ and a delta copy would speed these up...]

> Why do you think I use Linux ;=)
> Cygwin is an incredible program. It almost makes Windows useable. Kinda like
> a very bad unix.

> Opinions are my own and not necessarily representative of anyone sane.


Note: Not on the same spindle. The source and dest are different
physical disks. (if it matters, 500G sata disks.)

[IMHO, if a native win app can do it faster by at least an order of
magnitude, it's not going to be a "same-spindle" problem.]

I've looked at robocopy, but it doesn't appear to have any
mechanism to skip equal files and only copy files that have changed,
even if it copies the whole file instead of a delta. (perhaps
something could be rigged with the archive bit, though rdiff-backup
may not reliably handle the dos archive bit - it's something I'd
have to look at more.)

There are numerous sync programs available on Windows, even free
ones, but they don't do much in the way of logging etc, and I'd
prefer something that produces a log that can be saved and reviewed,
rather than just displaying some stuff on the screen that's mostly
useless except in an interactive session.

Thanks for the thoughts though

-Greg