can rsync scan files only with mtime since T?

[Ming Zhang]

On Fri, 2007-08-24 at 09:52 +1200, Darryl Dixon - Winterhouse Consulting
wrote:
> > Hi
> >
> > I have a file system that contains millions of small files. Since I
> > backup it everyday with rsync using slow WAN link, I think it will be
> > nice that if rsync can do this:
> >
> > An option that let rsync only check with remote rsync daemon about local
> > files that has last modification time newer than one day ago (so is
> > modified since yesterday backup). This can greatly reduce the WAN
> > traffic.
> >
> > Is this doable with current rsync?
> >
> 
> Hi Ming, List,
> 
> I thought I'd reply as I have used rsync in a similar scenario (~1TB of 13
> million files in two filesystems backed up offsite).
> 
> There are a couple of approaches that will do what you want - what OS are
> you using? (Windows, Solaris, Linux ...?). One is to run 'find -mtime -1 >

Linux,

> my_files.list' and then use the rsync --files-from=my_files.list to send
> only the new files. Running find can be time consuming(!), but effectively
> that's what you'd be doing with an 'rsync -mtime -1' option anyway.

rsync has such option? i do not know.

> Another option (and this is the one that I used) is to audit filesystem
> events as they are happenining, and keep a live list of all modified files
> all the time. This list can then be fed to rsync with the --files-from
> option.
> 
> On Solaris this can be achieved with the BSM module and NFS logging (if
> you're running and NFS server). On Linux I heavily modified pyinotify
> (http://pyinotify.sourceforge.net) to achieve the same result. The outcome
> is that every 5 minutes during the day I ship all the files changed in the
> previous 5 minutes offsite to the backup server. This works perfectly -
> and the volume of change is about 30,000 new files per day!

this sounds cool. i will look into this inotify. my last look at inotify
give me an impression that it can not scalable enough to observe a file
system with multiple million files. maybe i am seriously wrong.


just curious, how you deal with file/directory that were deleted or
renamed.


> 
> Email me direct if you want more details :)
> 
> regards,
> Darryl Dixon
> Winterhouse Consulting Ltd
> http://www.winterhouseconsulting.com
> darryl.dixon at winterhouseconsulting.com
> 
-- 
Ming Zhang


@#$%^ purging memory... (*!%
http://blackmagic02881.wordpress.com/
http://www.linkedin.com/in/blackmagic02881
--------------------------------------------