Transferring large amount of files w/ Rsync - Initial transfer

israel Gold rhglaw1 at barak.net.il
Mon Jan 24 07:47:17 GMT 2005


Here is what rsync man page states: 
-c, --checksum
    This forces the sender to checksum all files using a 128-bit MD4
checksum before transfer.    
    The checksum is then explicitly checked on the receiver and any
files of the same name 
    which already exist and have the same checksum and size on the
receiver are not 
    transferred. This option can be quite slow. 

This is too slow and would no do the job I have described.
I would like to complete the initial file transfer using tapes and
result with EXACTLY the same situation as if 
I did it using rsync. This means to create the checksums on the remote
site immediately after copying the files
from the tapes, and not during file synchronization.

Am I missing something ?

--
IsraeL Gold, WiGo Technologies, rhglaw1 at barak.net.il
--

-----Original Message-----
From: Jan-Benedict Glaw [mailto:jbglaw at lug-owl.de] 
Sent: Sunday, January 23, 2005 2:08 PM
To: israel Gold
Cc: rsync at lists.samba.org
Subject: Re: Transferring large amount of files w/ Rsync - Initial
transfer


On Sun, 2005-01-23 12:44:28 +0200, israel Gold <rhglaw1 at barak.net.il>
wrote in message <000001c50138$839cb9d0$8d1a4055 at il.sangate.com>:
> The following problem has appeared in the past but I have not seen 
> satisfactory solution. It is common when synchronizing large amount of

> files over slow link.

Should work, at least if your machine does have enough RAM to keep the
file list in memory...

> I would like to synchronize 1TB of data between local and remote sites

> over WAN. For the initial synchronization, I would like to transfer 
> the files from the local site to the remote site using tapes,
> and use Rsync only for the incremental synchronization.

No problem.

> After copying the files from the tapes at the remote site, it would 
> have been nice if Rsync had an option that would scan the files and 
> generate the checksums/hash values necessary for the incremental 
> synchronization step. (so that I would not need to use Rsync for the 
> initial transfer).

Have you searched the man page for "checksum"? I'm sure you didn't ...

> If this was an option, do I need to generate the checksums/hash values

> on the local site prior to incremental synchronization ?

See manpage :-)

> Is it complicated to add to Rsync an option that would just generate 
> checksums/hash values of files so that the solution above becomes 
> feasible ?.

No need for that...

MfG, JBG

-- 
Jan-Benedict Glaw       jbglaw at lug-owl.de    . +49-172-7608481
_ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen
Krieg  _ _ O
 fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM |
TCPA));



More information about the rsync mailing list