Rsync recursion

[tim.conway at philips.com]

I often use the -R option
example:
system1:/bigdir to system2:/backups/bigdir

list=bigdir/subdir/subdir...
cd /
rsync -HaRz listitem system2:/backups
that will put subdir/subdir as system2:/backups/bigdir/subdir/subdir
now, if you have items above that one, you've got more complexity, as you 
can't do bigdir -depth=1 or something like that.
what you can do is
rsync -HRz bigdir/* system2:/backups

What I actually did was to write a recursive shell script to process the 
output of find, giving only items with with less than some maximum number 
of subcomponents.  Of course, a file next to a directory that was an item 
would have no subcomponets at all, and would be sent alone.

The only lack in these processes is that you can't send over an empty 
directory.  The only way to get that is to have the empty directory be 
under another directory sent recursively.

Here's my splitter.
+++++++++++++++++++++++++++++++++++++++++++
#!/bin/sh

limit=$1
file=$2

splitdir(){

dir=$1

pathlength=`echo $dir |tr / ' '|wc -w`
pathlength=`echo $pathlength`

searchpat="^$dir/"
[ "$searchpat" = "^/" ] && searchpat='^'

grep $searchpat $file |
cut -d/ -f1-`expr $pathlength + 1` |
uniq -c |
while read dircount subdir
        do

                if [ "$dircount" -le "$limit" ]
                        then
                                echo $subdir
                        else
                                (splitdir $subdir)
                fi

        done

}

splitdir
+++++++++++++++++++++++++++++++++++++++++++++
Obviously, if you go a couple-hundred directories deep, it'll use a lot of 
resources.
file ($2) is a file made by find.
in the example above, limiting to 100000 items per rsync run:
cd /
find bigdir -print >listfile
splitter  100000 listfile
cat listfile |while read item
do
rsync -WHaRz $item system2:/backups
done

recursively sending a file is, sensibly, treated as sending the file 
itself.  The bad thing about breaking things like this up is that it makes 
the -H option less meaningful, since if you have multiple links to a file, 
but they are all done in seperate runs, the link relationship is lost.


Tim Conway
tim.conway at philips.com
303.682.4917 office, 303.921.0301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, caesupport2 on AIM
"There are some who call me.... Tim?"




"David Rasch" <rasch at raschnet.com>
Sent by: rsync-admin at lists.samba.org
07/30/2002 02:53 PM

 
        To:     <rsync at lists.samba.org>
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        RE: Rsync recursion
        Classification: 



> I'm trying to break up my rsync process by separating a directory tree
> into multiple rsync processes because I'm witnessing some errors trying
> to rsync large directory trees on Windows machines. After breaking up
> the tree I tried to rsync each individual directory starting from the
> bottom directory on up using the command:
>
>                foreach ($array as $directory){                 /*  
$array = list of
> directories gathered from the larger directory I'm trying to rsync */
>                                rsync -vz -e ssh 
username at hostname:/src/$directory/
> /dst/$directory/
>                }
>
To synchronize all the files in this directory, I believe you will need
the --recursive option.  This would work without the recursive option if
your 'foreach' statement iterated over all files in the tree.

Try this instead:

                 rsync -vrz -e ssh username at hostname:/src/$directory/ 
/dst/$directory/



Good luck,

David


-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html