sync 54 million files, tuning rsync?

[Jerry]

The design is outside of my control.  It's not a
regular replication it's a migration from one netapp
nfs appliance to another. (And no, I can't use the
wonderful block-level replication they provide due to
some other complex reasons).

I'm wondering if there is a way to tell rsync, go
ahead, take each dir off the top level and fire up an
independent rsync to do this.  It seems it's not using
all my cpu, memory, or bandwidth (on either side) to
accomplish the task.  I'm not sure where I'm being
throttled, except maybe the NFS client on the Solaris
box not being very efficient (Solaris 10, so it
*should* be).

--- Frank Hamersley <terabite at bigpond.com> wrote:

> > -----Original Message-----
> > From:
> rsync-bounces+terabite=bigpond.com at lists.samba.org
> > Behalf Of Dirk Schenkewitz
> > Sent: Tuesday, 14 February 2006 10:32 PM
> > To: rsync at lists.samba.org
> > Subject: Re: sync 54 million files, tuning rsync?
> (offlist)
> > 
> > Hi Jerry,
> > 
> > I reply offlist because I'm everything but an
> expert, so
> > you might get a better answer from someone else.
> > Anyway, perhaps it would be good if you post to
> the list 
> > what you're already doing.
> > 
> > On Tuesday, 14. February 2006 03:31, Jerry wrote:
> > > I'm trying to sync up 54 million files.  I can
> break
> > > it down into different applications, but I still
> have
> > > to accomplish 17 million files in one "chunk" if
> > > possible.
> > 
> [snip]
> Jerry,
> 
> <OutsideTheSquare>
> Given you have 54 million entries have you ever
> considered using a genuine DBMS (with replication)
> rather than pounding the filesystem to death?
> </OutsideTheSquare>
> 
> Cheers, Frank.
> 
> --
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com