How does rsync file verification work?
henri.shustak at gmail.com
Sat Feb 4 16:51:05 MST 2012
There are a number of data verification tools which may be options. One possibility is FingerPrint, a tool designed to verify data sets (not designed as a trip wire) : http://bit.ly/fingerprint-oriontransfer
Also another thing to keep in mind is that some file systems / hardware storage systems will perform checksums for data so that in the event that the hardware's ECC check is not able to resolve the problem the file system / hardware will detect the issue and attempt repair.
Finally, keep in mind that if you have a lot of data, this verification process may use considerable resources on your system(s).
Hope this helps.
This email is protected by LBackup, an open source backup solution.
> Rsync does not verify the data after it is written to the disk. The
> whole file checksum is only done to ensure that the file crossed the
> network intact.
> This is not unusual. Backup programs generally don't re-read from the
> target to confirm a successful write. In fact if they did they would
> probably just be reading from the cache anyway.
> If you are really paranoid about this then you need a separate file
> verification tool. Plus you need to make sure the cache is dumped
> between the rsync run and the file verification run.
> Note that the only time I have seen rsync write corrupt files was when
> a bad DIMM was corrupting data in the cache. The solution to that is
> ECC RAM. The only other way I have heard of it happening is a failing
> disk in the backup system and the solution to that is SMART monitoring.
>> I want to use rsync to create backups. I think I need a
>> verification that: 1) data is written to disk. 2) data is read back
>> from disk. 3) verify the checksum of data. Does the automatic file
>> verification in rsync works like this? Technically, I'd like to
>> know how does we make step 2 read data from disk and not from file
>> cache/buffer in OS? Or, do OS/harddrives always perform written
>> data verification anyway so step 2 is not really needed?
More information about the rsync