rsync based on checksum only

Dan Stromberg drsalists at
Sat Jul 7 18:34:50 MDT 2012

On Sat, Jul 7, 2012 at 4:15 PM, Henri Shustak <henri.shustak at>wrote:

> > Wow!  Thanks for making it so easy.  I will try that asap.
> If you do not have any luck with the patched version of rsync there are
> various projects which spring to mind which offer this kind of
> functionality.

For backups, not really for file transfers, there's my pet project,
It's been operating strictly on SHA hashes from the beginning.  It also
compresses its file chunks using xz.  xz is -=not=- blazing fast, but it
tends to produce nice, small files, and you generally only need to compress
a given file content (chunk, really - large files are divided into multiple
chunks) once, so subsequent backups are relatively quick.

In all honesty and candor, it has a little trouble on Windows (Cygwin)
because Windows will sometimes ENOENT on a read, but it's been working
great on *ix, including multiple Linuxes, *BSD, Solaris, OS/X and Haiku.
It's been hard to get myself to test on Windows.  :)

it also has a pretty comprehensive suite of automated tests and good
documentation, to help you feel more confident using it.

It doesn't use hardlinks at all in its repository, but it's very good at
detecting hard links in the files you're backing up - it doesn't just rely
on st_nlink, but instead uses a bloom filter.  So you can pretty easily
transfer a backshift repo to a new system (using rsync of course!), but you
aren't likely to need to as soon, because of the compression; even most
metadata is stored compressed.  IOW, the tradeoff is: your initial backups
are slow, but they're all small.

It needs a relatively capable filesystem for its repo, like ext4 or xfs.
ext3 and Netapp are known to not work because they can't handle the large
directory (one directory in particular grows to 2^20 files) backshift

The format it uses for its repo is custom, but it's pretty straightforward.

To do a save, you pipe find into it, like cpio.  To do a restore, you ask
it to generate a tar archive, and you pipe that to tar xvfp.  This means
you don't need to learn a unique file selection language, and it means the
software isn't subject to security issues during restore - or rather, no
issues that aren't taken care of by patches to your tar program.

HTH someone.  ^_^
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the rsync mailing list