Huge directory tree: Get files to sync via tools like sysdig

[Ben RUBSON]

> On 10 Feb 2017, at 01:21, Karl O. Pinc <kop at meme.com> wrote:
> 
> On Fri, 10 Feb 2017 12:38:32 +1300
> Henri Shustak <henri.shustak at gmail.com> wrote:
> 
>> As Ben mentioned, ZFS snapshots is one possible approach. Another
>> approach is to have a faster storage system. I have seen considerable
>> speed improvements with rsync on similar data sets by say upgrading
>> the storage sub system.
> 
> Another possibility could be to use lvm and lvmcache to throw a ssd in
> front of the spinning disks.  This would only improve things if
> you didn't otherwise fill up the cache with data -- you want
> the cache to contain inodes.  So this might work only if your
> ssd cache was larger than whatever amount of data you typically
> write between rsync runs, plus enough to hold all the inodes
> in your rsync-ed fs.
> 
> I've not tried this.  I'm not even certain it's a good idea.  It's
> just a thought.

It's also possible to have a SSD cache with ZFS (called the L2ARC).
You can even ask this cache to only store your metadata.

Some (same ?) changes may also be needed on receiver/server side too
(depending on its current setting) to see a performance improvement.

Ben