Transferring Large Files w/ Rsync - Initial Xfer

d c casteld73 at yahoo.com
Tue Jan 4 02:30:06 GMT 2005


Hmm---
 
Let me clarify a few things and apply it to your response to verify I understand.
 
I have 51 file ranging from 1mb to 1.9gb in size.  These files reside on a share of a windows server.  They are the result of an export from VMWare's ESX server.
 
I hooked up a usb drive to the windows box and copied the files to the drive.  Subsequently, I shipped the drive to a remote location.
 
At this location we copied those same files to a directory on a windows server which is shared.  Then- We point a freebsd server running rsync to this  windows share mounted with smb_fs.
 
At this point I ran rsync at the home location to sync the files between the two locations since some of the data has changed since it was copied and then shipped.
 
 IT appears that it needs to copy all the files over again...or is it possible that they have changed so much that it needs to do this.  But from what I understand rsync process the files on both sides and then determines what needs to be copied ?????
 
Note-  I am sending this data via ssh and connecting to the rsync daemon via a port forward 8730:localhost:873.  
 
Thanks for the quick reponses so far.  I have a feeling what I am trying to do is not feasible over T1's.
 
Anyone else able to move large files over the wan via encrypted ssh tunnels with rsync?

Wayne Davison <wayned at samba.org> wrote:
On Mon, Jan 03, 2005 at 04:40:03PM -0800, d c wrote:
> The source files have slightly chnaged and I now am trying to sync
> them. However, it seems that rsync has to copy the entire file
> itself the first time inorder to build the index (hash) needed in
> order to have more efficient transfers in the future.

The hash is determined at runtime. Rsync needs to read the whole file
on the receiving side (to generate the hash), the sender then uses that
hash data to send any unmatched data (by reading its version of the
file), and then the receiving side creates a duplicate file based on its
original one and the data from the sender. It then moves this new file
over the old one.

So, it's hard to know what you're referring to. You may just be seeing
the new temp file being created (which will always happen unless you use
the --inplace option, which is new for 2.6.3, but is only efficient for
things like log files that get appended data). You might be seeing a
new copy because you accidentally put the file into the wrong spot on
the recieving side (try sending a small file to the same directory as
the big file to test this). Or, the file may be compressed, in which
case a small change to the uncompressed data results in a massive change
in the compressed data (there is a patch available for gzip that makes
its compressed files more rsync friendly, if you need it).

..wayne..

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
HTML attachment scrubbed and removed


More information about the rsync mailing list