sync 54 million files, tuning rsync?

[Frank Hamersley]

G'day Jerry,

> -----Original Message-----
> From: Jerry [mailto:juanino at yahoo.com]
> Sent: Wednesday, 15 February 2006 2:55 PM
> To: Frank Hamersley; rsync at lists.samba.org
> Subject: RE: sync 54 million files, tuning rsync?
>
> The design is outside of my control.  It's not a
> regular replication it's a migration from one netapp
> nfs appliance to another. (And no, I can't use the
> wonderful block-level replication they provide due to
> some other complex reasons).

Is it a once of event, or to be repeated at intervals until least up until
the old netapp device is decommissioned?

> I'm wondering if there is a way to tell rsync, go
> ahead, take each dir off the top level and fire up an
> independent rsync to do this.  It seems it's not using
> all my cpu, memory, or bandwidth (on either side) to
> accomplish the task.  I'm not sure where I'm being
> throttled, except maybe the NFS client on the Solaris
> box not being very efficient (Solaris 10, so it
> *should* be).

Prolly (network) IO bound.

Just to confirm you are transferring data from 1 old appliance to another
new one using a (third) Solaris box to do the number crunching?  If so lots
of network latency for sure.  You could improve things by staging the files
on the Solaris box (has it enough DASD?) so it at least can do local IO for
the write side.

Then once the old netapp is decommissioned, perform another migration to the
new netapp - which will benefit from local IO on the read side.  If this is
to complicated a plan ... others may have brighter ideas!

> --- Frank Hamersley <terabite at bigpond.com> wrote:
>
> > > -----Original Message-----
> > > From:
> > rsync-bounces+terabite=bigpond.com at lists.samba.org
> > > Behalf Of Dirk Schenkewitz
> > > Sent: Tuesday, 14 February 2006 10:32 PM
> > > To: rsync at lists.samba.org
> > > Subject: Re: sync 54 million files, tuning rsync?
> > (offlist)
> > >
> > > Hi Jerry,
> > >
> > > I reply offlist because I'm everything but an
> > expert, so
> > > you might get a better answer from someone else.
> > > Anyway, perhaps it would be good if you post to
> > the list
> > > what you're already doing.
> > >
> > > On Tuesday, 14. February 2006 03:31, Jerry wrote:
> > > > I'm trying to sync up 54 million files.  I can
> > break
> > > > it down into different applications, but I still
> > have
> > > > to accomplish 17 million files in one "chunk" if
> > > > possible.
> > >
> > [snip]
> > Jerry,
> >
> > <OutsideTheSquare>
> > Given you have 54 million entries have you ever
> > considered using a genuine DBMS (with replication)
> > rather than pounding the filesystem to death?
> > </OutsideTheSquare>
> >
> > Cheers, Frank.
> >
> > --
> > To unsubscribe or change options:
> > https://lists.samba.org/mailman/listinfo/rsync
> > Before posting, read:
> > http://www.catb.org/~esr/faqs/smart-questions.html
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com