sync 54 million files, tuning rsync?

[Gian G. Spicuzza]

How about a shell script?  Scan the directories and start "x" instances 
of rsync at a time, when one finished, have the script fire up another, 
until the list of directories is complete......?

Gian

>
>The design is outside of my control.  It's not a
>regular replication it's a migration from one netapp
>nfs appliance to another. (And no, I can't use the
>wonderful block-level replication they provide due to
>some other complex reasons).
>
>I'm wondering if there is a way to tell rsync, go
>ahead, take each dir off the top level and fire up an
>independent rsync to do this.  It seems it's not using
>all my cpu, memory, or bandwidth (on either side) to
>accomplish the task.  I'm not sure where I'm being
>throttled, except maybe the NFS client on the Solaris
>box not being very efficient (Solaris 10, so it
>*should* be).
>
>--- Frank Hamersley <terabite at bigpond.com> wrote:
>
>  
>
>>>-----Original Message-----
>>>From:
>>>      
>>>
>>rsync-bounces+terabite=bigpond.com at lists.samba.org
>>    
>>
>>>Behalf Of Dirk Schenkewitz
>>>Sent: Tuesday, 14 February 2006 10:32 PM
>>>To: rsync at lists.samba.org
>>>Subject: Re: sync 54 million files, tuning rsync?
>>>      
>>>
>>(offlist)
>>    
>>
>>>Hi Jerry,
>>>
>>>I reply offlist because I'm everything but an
>>>      
>>>
>>expert, so
>>    
>>
>>>you might get a better answer from someone else.
>>>Anyway, perhaps it would be good if you post to
>>>      
>>>
>>the list 
>>    
>>
>>>what you're already doing.
>>>
>>>On Tuesday, 14. February 2006 03:31, Jerry wrote:
>>>      
>>>
>>>>I'm trying to sync up 54 million files.  I can
>>>>        
>>>>
>>break
>>    
>>
>>>>it down into different applications, but I still
>>>>        
>>>>
>>have
>>    
>>
>>>>to accomplish 17 million files in one "chunk" if
>>>>possible.
>>>>        
>>>>
>>[snip]
>>Jerry,
>>
>><OutsideTheSquare>
>>Given you have 54 million entries have you ever
>>considered using a genuine DBMS (with replication)
>>rather than pounding the filesystem to death?
>></OutsideTheSquare>
>>
>>Cheers, Frank.
>>
>>--
>>To unsubscribe or change options:
>>https://lists.samba.org/mailman/listinfo/rsync
>>Before posting, read:
>>http://www.catb.org/~esr/faqs/smart-questions.html
>>
>>    
>>
>
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam?  Yahoo! Mail has the best spam protection around 
>http://mail.yahoo.com 
>
>  
>
>------------------------------------------------------------------------
>
>_______________________________________
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>rsync mailing list
>rsync at lists.samba.org
>https://lists.samba.org/mailman/listinfo/rsync
>  
>