compression of source and target files
shachar at shemesh.biz
Sat Sep 22 07:36:14 GMT 2007
Kenneth Simpson wrote:
> Chuck Wolber wrote:
>> On Fri, 21 Sep 2007, Kenneth Simpson wrote:
>>> Hi - there's a flag for rsync to compress the files in transit - is it
>>> possible to compress one side (target) with gzip and have rsync still
>>> work correctly?
>> It'll still work correctly, but compressing a compressed file can actually
>> make it slightly bigger and wastes CPU cycles in the process.
> Sorry, I neglected to mention the source is uncompressed but
> we need to compress the target file because we're running out
> of disk space and the files are highly compressible.
> We can't compress the source since the files are large and
> compressing the source would create other problems.
> The original thought was to use a file system with compression (I
> think Linux has such a beast) but this would at least require a
> kernel rebuild which we won't be able to do for awhile.
> The second thought was that we might be able to gzip on the fly
> and have rsync work correctly (since it's compressing them in
gzip, as is, will destroy rsync's ability to sync partial file changes.
Gzip does have, however, a patch that adds a "rsyncable" option to the
command line, that makes the compressed output rsync ready. The only
problem I see with your suggestion is that, as far as I know, rsync
cannot sync a stream of data to a file. Do have a look at librsync,
however, which reportedly can do that.
More information about the rsync