Feature Request - Recursive Rsync Parameter - Example Script

jw schultz jw at pegasys.ws
Wed Oct 22 11:46:16 EST 2003


On Tue, Oct 21, 2003 at 06:25:51PM -0700, Matt Olson wrote:
> I wanted to flag a problem and offer a possible solution.
> 
> The problem:
> 
> large rsync operation fails on machines with modest amounts of memory.
> 
> Proposal:
> 
> Add a parameter to rsync recursive to specify a recursion level (See 
> example bash wrapper below).  (works with recursive file system rsyncs 
> only, i.e. -a or -r) The logic goes:

Limiting the depth of recursion is already supported just
not intuitive.

	rsync -r --exclude='/*/*/*/'

Your idea for a shell script to automate picking up the
lower levels is good and could compose the --exclude
pattern.  The next step would be to set the job partition
level based on path count as in "find $subtree -print|wc -l".

> if recursion switch true and recursion_level > 0
> 
>   -rsync this directory only
>   -call rsync for each subdirectory with a decremented recursion_level and 
> pass the same switches along
> 
> else (recursion_level really is 0)
> 
>   -perform the full rsync (from this level)
>   
> What this does is help break up the job into smaller pieces.  Otherwise 
> rsync can consume hundreds of megabyte of memory attempting to perform a 
> single operation.  In this scenario you'll see one rsync process for each 
> level of recursion.
> 
> Here's and example bash script that is an attempt at this idea:  (it 
> supports the -n options so you can see the calls it makes)
> 
> My bash scripting skills need some work, but, you get the idea.  If 
> someone wants to further develop this script, feel free.  
> 
> Cheers.
> 
> 
> #!/bin/bash
[snip]



More information about the rsync mailing list