RSYNC via pipe/socket ?

[Matt McCutchen]

On Fri, 2005-12-09 at 00:46 +0100, roland wrote:
> I`m trying to find a way to use lzo compression for the data being
> transferred by rsync.

It's easy set this up.  Consider this (for gzip but it's easy to do the
same for any compression program):

/usr/local/bin/gzip-wrap:
	#!/bin/bash
	gzip | "$@" | gunzip

/usr/local/bin/gunzip-wrap:
	#!/bin/bash
	gunzip | "$@" | gzip

Then run:
	rsync --rsh='gzip-wrap ssh' --rsync-path='gunzip-wrap rsync'
		<options source dest>

As elegant as this technique is, it fails because compression programs
perform internal buffering.  One rsync will send a block of data and
wait for an acknowledgement that the other side has received it, but
since the end of the data is buffered in the compression program, the
other side never responds and deadlock results.  There might be a way
around this, but I can't think of one.

The next best thing is to use rsync to generate a batch file on the
sender, bring the batch file to the receiver by hand (compressing and
decompressing), and apply the batch file on the receiver using rsync.
This effectively lets you compress the mainstream file-data part of the
transmission, but much of the metadata must still go over the wire
during the creation of the batch file.  See the man page for details.
-- 
Matt McCutchen
hashproduct at verizon.net
http://mysite.verizon.net/hashproduct/