rsync of STDIN to a file.

Matt McCutchen matt at mattmccutchen.net
Thu Nov 19 09:53:30 MST 2009


On Thu, 2009-11-19 at 16:28 +0000, Mark Young wrote:
> I tried an experiment to see how rsync coped with the tar compressed
> files versus the uncompressed files. I took a .tgz from last week and
> rsync'd it with last nights version. rsync achieved the transfer with
> a 1.61 speed up. Specifically 172MB of the 262MB file was transferred.
> I then tried the same thing with the uncompressed tar files. rsync
> achieved the transfer with a 690 speed up. Specifically 5MB for the
> 3.6GB file was transferred. Clearly if diskspace were not an issue
> this would be by far the superior option. Obviously I could increase
> the complexity of the overall solution with compress & uncompress jobs
> in sync with the backup strategy, but I believe in keeping it as
> simple as possible when it comes to backups.

Right.  The gzip --rsyncable option is designed for this case: files
compressed with --rsyncable should delta-transfer almost as well as
uncompressed files, at a minor penalty in the compression ratio.

> rdiff is a new one on me. Very interesting and something I'll
> experiment with. For our strategy of 7 daily backups and 12 monthly
> backups I guess I'd need to use two rdiff destinations so I could
> apply the --remove-older-than 1W and --remove-older-than 365D options
> respectively. It might not suit my current situation as explained
> above regarding storing a tarball versus a directory hierachy, but
> it's definitely worth knowing about.

I was talking about rdiff (a command-line interface to the
delta-transfer algorithm), not rdiff-backup (the incremental backup
tool).

> Also I'd not heard of the --rsyncable option for tar. It's not there
> on my Debian Etch system, nor in my cygwin tar. So I guess it's not in
> common distributions by default. Still it sounds useful and is
> something I will bear in mind as something to consider.

The option applies to gzip, not tar.  In fact, it looks like Fedora
applies an updated version of the gzip-rsyncable patch to their gzip
package.  You can get their patch from:

http://cvs.fedoraproject.org/viewvc/rpms/gzip/F-11/gzip-1.3.9-rsync.patch?view=co

-- 
Matt



More information about the rsync mailing list