Rsync recursion

Mark de Jong dejongm at
Wed Jul 31 10:26:02 EST 2002

I posted a question on the list entitled "Rsync windows to linux is
hanging" on 7/16 concerning rsync over ssh ( ). Have
you experienced, or have you heard of anyone experiencing, rsync hanging
using "-e ssh" when transfering from a windows machine to a linux
machine? I just recently did the same transfer using rsync in daemon
mode on the windows 2000 machine and the process finished flawlessly.
What's the difference besides the encrypted tunnel? Do you have any idea
as to why it would fail and why only from Windows?

Thanks for your time!

Mark J. de Jong
Senior Network Engineer - Secure Dog Hosting, Inc.
P.703.256.2869 F.703.256.3810 C.571.212.0027

-----Original Message-----
From: rsync-admin at [mailto:rsync-admin at]
On Behalf Of tim.conway at
Sent: Tuesday, July 30, 2002 6:31 PM
To: David Rasch
Cc: rsync at
Subject: RE: Rsync recursion

I often use the -R option
system1:/bigdir to system2:/backups/bigdir

cd /
rsync -HaRz listitem system2:/backups
that will put subdir/subdir as system2:/backups/bigdir/subdir/subdir
now, if you have items above that one, you've got more complexity, as
can't do bigdir -depth=1 or something like that.
what you can do is
rsync -HRz bigdir/* system2:/backups

What I actually did was to write a recursive shell script to process the

output of find, giving only items with with less than some maximum
of subcomponents.  Of course, a file next to a directory that was an
would have no subcomponets at all, and would be sent alone.

The only lack in these processes is that you can't send over an empty 
directory.  The only way to get that is to have the empty directory be 
under another directory sent recursively.

Here's my splitter.




pathlength=`echo $dir |tr / ' '|wc -w`
pathlength=`echo $pathlength`

[ "$searchpat" = "^/" ] && searchpat='^'

grep $searchpat $file |
cut -d/ -f1-`expr $pathlength + 1` |
uniq -c |
while read dircount subdir

                if [ "$dircount" -le "$limit" ]
                                echo $subdir
                                (splitdir $subdir)



Obviously, if you go a couple-hundred directories deep, it'll use a lot
file ($2) is a file made by find.
in the example above, limiting to 100000 items per rsync run: cd / find
bigdir -print >listfile splitter  100000 listfile cat listfile |while
read item do rsync -WHaRz $item system2:/backups done

recursively sending a file is, sensibly, treated as sending the file 
itself.  The bad thing about breaking things like this up is that it
the -H option less meaningful, since if you have multiple links to a
but they are all done in seperate runs, the link relationship is lost.

Tim Conway
tim.conway at
303.682.4917 office, 303.921.0301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, caesupport2 on AIM "There
are some who call me.... Tim?"

"David Rasch" <rasch at>
Sent by: rsync-admin at
07/30/2002 02:53 PM

        To:     <rsync at>
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        RE: Rsync recursion

> I'm trying to break up my rsync process by separating a directory tree

> into multiple rsync processes because I'm witnessing some errors 
> trying to rsync large directory trees on Windows machines. After 
> breaking up the tree I tried to rsync each individual directory 
> starting from the bottom directory on up using the command:
>                foreach ($array as $directory){                 /*  
$array = list of
> directories gathered from the larger directory I'm trying to rsync */
>                                rsync -vz -e ssh
username at hostname:/src/$directory/
> /dst/$directory/
>                }
To synchronize all the files in this directory, I believe you will need
the --recursive option.  This would work without the recursive option if
your 'foreach' statement iterated over all files in the tree.

Try this instead:

                 rsync -vrz -e ssh username at hostname:/src/$directory/ 

Good luck,


To unsubscribe or change options:
Before posting, read:

To unsubscribe or change options:
Before posting, read:

More information about the rsync mailing list