Huge directory tree: Get files to sync via tools like sysdig
Ben RUBSON
ben.rubson at gmail.com
Fri Feb 10 06:49:52 UTC 2017
> On 10 Feb 2017, at 01:21, Karl O. Pinc <kop at meme.com> wrote:
>
> On Fri, 10 Feb 2017 12:38:32 +1300
> Henri Shustak <henri.shustak at gmail.com> wrote:
>
>> As Ben mentioned, ZFS snapshots is one possible approach. Another
>> approach is to have a faster storage system. I have seen considerable
>> speed improvements with rsync on similar data sets by say upgrading
>> the storage sub system.
>
> Another possibility could be to use lvm and lvmcache to throw a ssd in
> front of the spinning disks. This would only improve things if
> you didn't otherwise fill up the cache with data -- you want
> the cache to contain inodes. So this might work only if your
> ssd cache was larger than whatever amount of data you typically
> write between rsync runs, plus enough to hold all the inodes
> in your rsync-ed fs.
>
> I've not tried this. I'm not even certain it's a good idea. It's
> just a thought.
It's also possible to have a SSD cache with ZFS (called the L2ARC).
You can even ask this cache to only store your metadata.
Some (same ?) changes may also be needed on receiver/server side too
(depending on its current setting) to see a performance improvement.
Ben
More information about the rsync
mailing list