large number of small files to move

Matt McCutchen matt at
Wed Feb 27 17:14:34 GMT 2008

On Mon, 2008-02-25 at 14:11 -0600, Terry wrote:
> I have a large migration of a single system going from windows to
> linux.  We are moving 9 TB and 60 million files.  I am planning on
> using rsync to move this data.  Are there any special considerations I
> should have in mind when moving this much data and more importantly,
> this many small files?
> I am mounting the volumes out of fstab like this:
> //dss01/f$    /mnt/dss01/f   cifs credentials=/etc/asdf,ro 0 0
> And rsync command is:
> rsync -rt --delete --stats /mnt/dss01/f /data00

In short: use rsync 3.0.0 (or a prerelease of it).  The incremental
recursion mode will make processing such an enormous number of files
much faster and will keep the memory usage reasonable.


More information about the rsync mailing list