rsync --link-dest and --files-from lead by a "change list" from some file system audit tool (Was: Re: cut-off time for rsync ?)

[Ken Chase]

inotifywatch or equiv, there's FSM stuff (filesystem monitor) as well.

constantData had a product we used years ago - a kernel module that dumped
out a list of any changed files out some /proc or /dev/* device and they
had a whole toolset that ate the list (into some db) and played it out
as it constantly tried to keep up with replication to a target (kinda like
drdb but async). They got eaten by some large backup company and the product
was later priced at 5x what we had paid for it (in the mid $x000s/y)

This 2003-4 technolog is certainly available in some format now.

If you only copy the changes, you're likely saving a lot of time.

/kc


On Mon, Jul 13, 2015 at 01:53:43PM +0000, Andrew Gideon said:
  >On Mon, 13 Jul 2015 02:19:23 +0000, Andrew Gideon wrote:
  >
  >> Look at tools like inotifywait, auditd, or kfsmd to see what's easily
  >> available to you and what best fits your needs.
  >> 
  >> [Though I'd also be surprised if nobody has fed audit information into
  >> rsync before; your need doesn't seem all that unusual given ever-growing
  >> disk storage.]
  >
  >I wanted to take this a bit further.  I've thought, on and off, about 
  >this for a while and I always get stuck.
  >
  >I use rsync with --link-desk as a backup tool.  For various reasons, this 
  >is not something I want to give up.  But, esp. for some very large file 
  >systems, doing something that avoids the scan would be desirable.
  >
  >I should also add that I mistrust time-stamp, and even time-stamp+file-
  >size, mechanism for detecting changes.  Checksums, on the other hand, are 
  >prohibitively expensive for backup of large file systems.
  >
  >These both bring me to the idea of using some file system auditing 
  >mechanism to drive - perhaps with an --include-from or --files-from - 
  >what rsync moves.
  >
  >Where I get stuck is that I cannot envision how I can provide rsync with 
  >a limited list of files to move that doesn't deny the benefit of --link-
  >dest: a complete snapshot of the old file system via [hard] links into a 
  >prior snapshot for those files that are unchanged.
  >
  >Has anyone done something of this sort?  I'd thought of preceding the 
  >rsync with a "cp -Rl" on the destination from the old snapshot to the new 
  >snapshot, but I still think that this will break in the face of hard 
  >links (to a file not in the --files-from list) or a change to file 
  >attributes (ie. a chmod would effect the copy of a file in the old 
  >snapshot).
  >
  >Thanks...
  >
  >	Andrew
  >
  >-- 
  >Please use reply-all for most replies to avoid omitting the mailing list.
  >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
  >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Ken Chase - ken at heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.