Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig

[Thomas Güttler]

Am 09.02.2017 um 16:21 schrieb Ben RUBSON:
>
>> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
>>
>> Am 09.02.2017 um 11:05 schrieb Ben RUBSON:
>>>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> we have a huge directory tree.
>>>>
>>>>
>>>> * 17M files (number of files)
>>>> * 2.2TBytes of data.
>>>> * Only 0.1% changes per day
>>>>
>>>> Current pain: rsyncs directory tree traversal needs to long to discover the changed files.
>>>
>>> Hi,
>>>
>>> On which type of FS is this directory ?
>>
>> ext4
>
> Any way to prefer snapshots in your backup strategy ?
> Or to use a ZFS ready OS to benefit from a SSD cache (which would store your metadata) ?

Yes, I think rsync is coming to the edge of its capabilities here. I guess a different strategy is needed.

I see these alternatives to rsync:

  - Incremental Snapshots at block-level device is one of them.
  - We get the application ported to access a storage server, and not file server any more.
  - ....

Do you see other alternatives?

Regards,
   Thomas Güttler



-- 
Thomas Guettler http://www.thomas-guettler.de/