Professional grade rsync?

[Jim Salter]

Silly suggestion, perhaps, but...

Ever considered simply breaking your sync down into several separate 
sets?  IE instead of rsync /foo/baz user at machine:/foo, where /foo/baz 
contains ten directories each then spiraling down into hundreds of 
thousands more... rsync /foo/baz/1 user at machine:/foo/baz, then rsync 
/foo/baz/2 user at machine:/foo/baz, and so on.  Enterprising sorts can, of 
course, write wrappers in Perl or the shell of your choice to 
automatically sync everything in /foo/baz one directory and/or file at a 
time with ease.

I haven't personally run into your upper end limits on heavy-duty 
equipment, but I've used the technique described above to keep old 32MB 
and 64MB machines in successful service as rsync servers; the principle 
is the same - scale your single sync down to a level that your hardware 
can handle.

(It should also be possible to make rsync more memory-friendly when the 
size of the job exceeds the resources available, by causing it to do the 
exact same thing as the wrapper described above does, but I'm not going 
to kvetch.)

Jim Salter
JRS Systems

> Hi, folks.
> 
> We've gone where no man has gone before.  On HP-UX, rsync bombs at about
> 1.75 million directories, files and links (combined sum) in a single
> transfer.
> 
> Is there a professional-grade alternative on HP-UX for folks willing to pay
> for it?  It wouldn't even need to be network-aware, just from single-system
> areas to the same box, but with the nifty delete and update features that
> rsync has.  My searches turn up unison and some other tools (BSD mirror,
> etc.), but rsync has beaten any other open-source solution hands down on the
> scalability side of things.  Now, we need more ...
> 
> Thanks,
> 
> A. Daniel King, System Analyst
> HP-UX, Linux, Solaris