rsync practical size limit?

[Crowder, Mark]

David,

I've used rsync to transfer multiple terabytes when replacing or
consolidating storage.  The practical limit is not the overall size, but
there are a couple that will crop up:

1.        This is alluded to in the FAQ, but the relationship between
amount of memory on the hosts and the number of files per sync is
important.  Rsync will use ~100 bytes per file on both the source and
destination servers, so if you are moving many millions of files you may
have to break it up into smaller bites - i.e. the "script that writes
other scripts" sort of technique.  No real problem here, unless the
files are all in one directory ...

2.        If the individual files are much larger than 1GB, rsync will
tend to copy the entire file over again rather than the differences.
This pretty much negates the benefits of rsync.  Luckily, most files
that large are database files and they have their own hot migration
techniques available.  

It's a VERY useful tool, though.

Regards,

Mark Crowder        

Texas Instruments, WW Make IT Infrastructure

"Ninety percent of the time things turn out worse than 

you thought they would.  The other ten percent of the 

time you had no right to expect that  much."

                     Augustine

 

 

-----Original Message-----
From: rsync-bounces+m-crowder=ti.com at lists.samba.org
[mailto:rsync-bounces+m-crowder=ti.com at lists.samba.org] On Behalf Of
David Saunders
Sent: Monday, March 21, 2005 2:54 PM
To: rsync at lists.samba.org
Subject: FW: rsync practical size limit?

 

I'm a heavy user of rsync for what sometimes amounts to more than 175GB
of

data.  Is there a practical size limit for rsync?

 

-- 

To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-------------- next part --------------
HTML attachment scrubbed and removed