Parallel rsync's for better Performance.

Matthias Schniedermeyer ms at
Wed Oct 28 03:01:17 MDT 2009

On 28.10.2009 09:05, Satish Shukla wrote:
> Hi ,
> We have huge data to sync usually everyday and I wish rsync could guarantee performance.
> I thought of spliting the directories and run parallel rsyncs on them. It may cost me some network, but I can control that from the MAX_RSYNC_PROCESS variable. Can some one evaluate pros and cons of this design?. Any help is heartily appreciated. 

That only works IF:
- You have SSDs (preferably good ones, both sides)
- Each rsync covers a different physical HDD (both sides)
- You have a massive Array with truck-loads of HDDs and a matching
  controller or something along that line (again both sides).
- A combination of the above would also work

Otherwise parallel rsyncs completly kill any performance you had because 
normal HDDs will fall into a seek-storm, when more than 1 rsync works on 

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

More information about the rsync mailing list