large number of small files to move

[Matt McCutchen]

On Mon, 2008-02-25 at 14:11 -0600, Terry wrote:
> I have a large migration of a single system going from windows to
> linux.  We are moving 9 TB and 60 million files.  I am planning on
> using rsync to move this data.  Are there any special considerations I
> should have in mind when moving this much data and more importantly,
> this many small files?
> 
> I am mounting the volumes out of fstab like this:
> //dss01/f$    /mnt/dss01/f   cifs credentials=/etc/asdf,ro 0 0
> 
> And rsync command is:
> rsync -rt --delete --stats /mnt/dss01/f /data00

In short: use rsync 3.0.0 (or a prerelease of it).  The incremental
recursion mode will make processing such an enormous number of files
much faster and will keep the memory usage reasonable.

Matt