Syncing large amounts of data

[Steve Bonds]

A good solution for the "lots of little files" problem is syncing below
the filesystem layer (usually either at the logical volume or disk
level.)  This allows you to avoid the stat() system call on loads of files
while looking for the ones to send.

As was already suggested, if you can avoid the search by having your
application keep track of what has changed and what has not, then you can
save yourself a lot of system impact.

  -- Steve

On Wed, 12 Feb 2003, Adam Herbert aherbert-at-franklinamerican.com |Rsync List| wrote:

> I need some suggestions. Here's my setup:
> 
> 	800GB of Data
> 	14,000,000+ Files
>       No changes just additions
>       Files range in size from 30k - 190k