Huge directory tree: Get files to sync via tools like sysdig

[Steven Levine]

In <E48BA632-D8D5-42AF-AE67-17438C201AC5 at gmail.com>, on 02/10/17
   at 12:38 PM, Henri Shustak <henri.shustak at gmail.com> said:

>As Ben mentioned, ZFS snapshots is one possible approach. Another
>approach is to have a faster storage system. I have seen considerable
>speed improvements with rsync on similar data sets by say upgrading the
>storage sub system.

This is true.  In addition different file systems has different
performance wrt stat().

A lot depends on what kind of backup that is required.  If a full backup
that is accurate to a point in time is required, then something like ZFS
makes sense.  If the system is servers that do in memory cachinng, there
will probably be a need to ensure that their on-disk state is consistent
before the snapshot is taken.

If the only requirment is to ensure that everything known to have changed
on disk is backed up, the event based solution will avoid the high cost of
stat-ing every file.

It might be interesting to evaluate a mixed solution.  Use events to track
directory changes and let rsync sort out what to do for each directory.

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------