Huge directory tree: Get files to sync via tools like sysdig

[Axel Kittenberger]

> Has someone experience with collecting the changed files
> with a third party tool which detects which files were changed?

I don't know of sysdig but am the developer of Lsyncd which does exactly
that, collect file changes via inotify event mechanism and then calls rsync
with a matching filter mask.

However, since you say, your directory tree is hugh, the main issue is that
for every directory an inotify watch must be created, taking about 1KB of
kernel memory per watch. If you got a million directories this is a GB of
unswapable memory use.

Unfortunally the Linux kernel doesn't provide a better way yet, and I
suppose other tools like sysdig suffer from the same issue. There is
fanotify, but that doesn't report move event and thus is not useable for
this task.

Kind regards, Axel

On Thu, Feb 9, 2017 at 10:05 AM, Thomas Güttler <
guettliml at thomas-guettler.de> wrote:

> Hi,
>
> we have a huge directory tree.
>
>
>  * 17M files (number of files)
>  * 2.2TBytes of data.
>  * Only 0.1% changes per day
>
> Current pain: rsyncs directory tree traversal needs to long to discover
> the changed files. Only few files change.
>
> I discovered the tool sysdig which could be used to monitor the files
> which were changed.
>
> Then we could feed the list of changed files to rsync and avoid the long
> directory traversal of rsync.
>
> Has someone experience with collecting the changed files with a third
> party tool which detects which
> files were changed?
>
> Regards,
>  Thomas Güttler
>
>
>
> --
> Thomas Guettler http://www.thomas-guettler.de/
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailma
> n/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20170209/6e53dfd0/attachment.html>