Nice little performance improvement

Matt McCutchen matt at mattmccutchen.net
Thu Oct 15 21:26:42 MDT 2009


On Thu, 2009-10-15 at 19:07 -0700, Mike Connell wrote:
> Today I tried the following:
>  
> For all subsub directories
>     a) Fork a "du -s subsubdirectory" on the destination
> subsubdirectory
>     b) Run rsync on the subsubdirectory
>     c) repeat untill done
>  
> Seems to have improved the time it takes by about 25-30%. It looks
> like the du can
> run ahead of the rsync...so that while rsync is building its file
> list, the du is warming up
> the file cache on the destination. Then when rsync looks to see what
> it needs to do
> on the destination, it can do this more efficiently.

Interesting.  If you're not using incremental recursion (the default in
rsync >= 3.0.0), I can see that the "du" would help by forcing the
destination I/O to overlap the file-list building in time.  But with
incremental recursion, the "du" shouldn't be necessary because rsync
actually overlaps the checking of destination files with the file-list
building on the source.

-- 
Matt



More information about the rsync mailing list