Rsync recursion

[Mark de Jong]

Tim,
I posted a question on the list entitled "Rsync windows to linux is
hanging" on 7/16 concerning rsync over ssh (
http://www.mail-archive.com/rsync@lists.samba.org/msg04596.html ). Have
you experienced, or have you heard of anyone experiencing, rsync hanging
using "-e ssh" when transfering from a windows machine to a linux
machine? I just recently did the same transfer using rsync in daemon
mode on the windows 2000 machine and the process finished flawlessly.
What's the difference besides the encrypted tunnel? Do you have any idea
as to why it would fail and why only from Windows?

Thanks for your time!

Mark J. de Jong
,.,.,.,...,.,,.,..,.,....,.,..,.,..,.,.,,.,...,..,,...
Senior Network Engineer - Secure Dog Hosting, Inc.
P.703.256.2869 F.703.256.3810 C.571.212.0027
http://www.secdog.com



-----Original Message-----
From: rsync-admin at lists.samba.org [mailto:rsync-admin at lists.samba.org]
On Behalf Of tim.conway at philips.com
Sent: Tuesday, July 30, 2002 6:31 PM
To: David Rasch
Cc: rsync at lists.samba.org
Subject: RE: Rsync recursion


I often use the -R option
example:
system1:/bigdir to system2:/backups/bigdir

list=bigdir/subdir/subdir...
cd /
rsync -HaRz listitem system2:/backups
that will put subdir/subdir as system2:/backups/bigdir/subdir/subdir
now, if you have items above that one, you've got more complexity, as
you 
can't do bigdir -depth=1 or something like that.
what you can do is
rsync -HRz bigdir/* system2:/backups

What I actually did was to write a recursive shell script to process the

output of find, giving only items with with less than some maximum
number 
of subcomponents.  Of course, a file next to a directory that was an
item 
would have no subcomponets at all, and would be sent alone.

The only lack in these processes is that you can't send over an empty 
directory.  The only way to get that is to have the empty directory be 
under another directory sent recursively.

Here's my splitter.
+++++++++++++++++++++++++++++++++++++++++++
#!/bin/sh

limit=$1
file=$2

splitdir(){

dir=$1

pathlength=`echo $dir |tr / ' '|wc -w`
pathlength=`echo $pathlength`

searchpat="^$dir/"
[ "$searchpat" = "^/" ] && searchpat='^'

grep $searchpat $file |
cut -d/ -f1-`expr $pathlength + 1` |
uniq -c |
while read dircount subdir
        do

                if [ "$dircount" -le "$limit" ]
                        then
                                echo $subdir
                        else
                                (splitdir $subdir)
                fi

        done

}

splitdir
+++++++++++++++++++++++++++++++++++++++++++++
Obviously, if you go a couple-hundred directories deep, it'll use a lot
of 
resources.
file ($2) is a file made by find.
in the example above, limiting to 100000 items per rsync run: cd / find
bigdir -print >listfile splitter  100000 listfile cat listfile |while
read item do rsync -WHaRz $item system2:/backups done

recursively sending a file is, sensibly, treated as sending the file 
itself.  The bad thing about breaking things like this up is that it
makes 
the -H option less meaningful, since if you have multiple links to a
file, 
but they are all done in seperate runs, the link relationship is lost.


Tim Conway
tim.conway at philips.com
303.682.4917 office, 303.921.0301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, caesupport2 on AIM "There
are some who call me.... Tim?"




"David Rasch" <rasch at raschnet.com>
Sent by: rsync-admin at lists.samba.org
07/30/2002 02:53 PM

 
        To:     <rsync at lists.samba.org>
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        RE: Rsync recursion
        Classification: 



> I'm trying to break up my rsync process by separating a directory tree

> into multiple rsync processes because I'm witnessing some errors 
> trying to rsync large directory trees on Windows machines. After 
> breaking up the tree I tried to rsync each individual directory 
> starting from the bottom directory on up using the command:
>
>                foreach ($array as $directory){                 /*  
$array = list of
> directories gathered from the larger directory I'm trying to rsync */
>                                rsync -vz -e ssh
username at hostname:/src/$directory/
> /dst/$directory/
>                }
>
To synchronize all the files in this directory, I believe you will need
the --recursive option.  This would work without the recursive option if
your 'foreach' statement iterated over all files in the tree.

Try this instead:

                 rsync -vrz -e ssh username at hostname:/src/$directory/ 
/dst/$directory/



Good luck,

David


-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.tuxedo.org/~esr/faqs/smart-questions.html




-- 
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.tuxedo.org/~esr/faqs/smart-questions.html