Data corruption check

[Keith Lofstrom]

On 9/18/07, Fabian Cenedese <Cenedese at indel.ch> wrote:
> I was wondering what happens if a file that is regularly synched but
> seldom changes gets corrupted in the copy.

On Wed, Sep 19, 2007 at 09:23:28AM +0200, Fabian Cenedese wrote:
> I was asking because I'm responsible for our backups. The
> current solution with rsync works nicely. While the RAID storage
> also monitor the HD's SMART state I was still wondering
> about a way to detect otherwise unknown data corruption.

I run rsync inside of dirvish (www.dirvish.org) for automated
backups.  I also run osiris (osiris.shmoo.com) which scans for
modified files, both checking the metadata and a hash of the 
actual data, finding all changes relative to a database on a
central osiris server.  It should be possible to combine these
mechanisms, scanning for changes with osiris in parallel on all
the backup clients, then using rsync to move the files that
osiris detected as changed, without rsync having to scan the
whole filesystem again.  Or some combination of both, letting
osiris look in detail at high-vulnerability files daily, then
the rest of the filesystem in sections over the course of a week.

The osiris scheduler is weird;  it is designed to be robust against
system vandals, but difficult to configure and especially difficult
to run as part of a larger app.  But a good programmer would be
able to break out the scanning components and tie them into a
different tool.

In any case, if you are worried about file corruption, consider 
running osiris, which will tell you more than you ever wanted to
know about what is changing in your filesystems.

Keith

-- 
Keith Lofstrom          keithl at keithl.com         Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs