Feature Request - Recursive Rsync Parameter - Example Script
jw at pegasys.ws
Wed Oct 22 11:46:16 EST 2003
On Tue, Oct 21, 2003 at 06:25:51PM -0700, Matt Olson wrote:
> I wanted to flag a problem and offer a possible solution.
> The problem:
> large rsync operation fails on machines with modest amounts of memory.
> Add a parameter to rsync recursive to specify a recursion level (See
> example bash wrapper below). (works with recursive file system rsyncs
> only, i.e. -a or -r) The logic goes:
Limiting the depth of recursion is already supported just
rsync -r --exclude='/*/*/*/'
Your idea for a shell script to automate picking up the
lower levels is good and could compose the --exclude
pattern. The next step would be to set the job partition
level based on path count as in "find $subtree -print|wc -l".
> if recursion switch true and recursion_level > 0
> -rsync this directory only
> -call rsync for each subdirectory with a decremented recursion_level and
> pass the same switches along
> else (recursion_level really is 0)
> -perform the full rsync (from this level)
> What this does is help break up the job into smaller pieces. Otherwise
> rsync can consume hundreds of megabyte of memory attempting to perform a
> single operation. In this scenario you'll see one rsync process for each
> level of recursion.
> Here's and example bash script that is an attempt at this idea: (it
> supports the -n options so you can see the calls it makes)
> My bash scripting skills need some work, but, you get the idea. If
> someone wants to further develop this script, feel free.
More information about the rsync