Rsyncing a very large directory tree (over 50,000 files)

[Matt McCutchen]

On Wed, 2006-06-14 at 19:07 +0100, Andrew Hodgson wrote:
> Is there anything I need to be aware of before doing this?  I started
> the script this morning, but it was still building the file list after
> around 15 minutes.  Is it better to do it using several points, then
> when I have the structure on the other machine I can then do the whole
> tree in one go?  Will the complete file list need to be sent across
> each time I run the program?

Yes, rsync will send the complete file list each time it runs.  It seems
odd to me that building the file list would take 15 minutes; when I back
up the system partition of my computer (300,000 files) rsync takes
perhaps 5 minutes to build the file list.  I don't think using several
points would be better or worse than doing it all at once, just more
complicated.
> 
> - /bin/
> + /binbackup/bin/
> - /binbackup/**
> I want to exclude the bin directory and anything in the binbackup
> directory apart from the bin directory within.

The third filter will match /binbackup/bin/foo and cause it to be
excluded, which isn't what you want.

You can do this:

- /bin/
+ /binbackup/bin/
- /binbackup/*
> 
That way, the third filter does not
exclude /binbackup/bin/foo.  /binbackup/bar/baz is technically included,
but since /binbackup/bar is excluded, rsync never even goes inside it.

Or you can do this:

- /bin/
+ /binbackup/bin/***
- /binbackup/**

The second filter overrides the third one and causes /binbackup/bin and
everything inside to be included.

Matt