rsync of STDIN to a file.

Mark Young mark_young at hotmail.com
Thu Nov 19 10:29:21 MST 2009


Ok thanks Matt,

Ah. I think I've realised my mistake. I saw this webpage http://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/ which showed the following command.
  GZIP="--rsyncable" tar zcvf toto.tar.gz /toto

I've just realised that gzip must be picking up the environment variable when it's invoked by tar. Cool.

So if I understand you correctly you are suggesting that I modify my backup line to be something like this:

tar czf - --files-from $FILE_LIST | gzip --rsyncable > ${BACKUP_FILE}
or:
GZIP="--rsyncable" tar cvf - --files-from $FILE_LIST > ${BACKUP_FILE}


then
rsync ${BACKUP_FILE} $HOST:${BACKUP_FILE}

I see that Debian's gzip has --rsycnable available. Hoorah.

I just tried some tests but didn't get the results I expected. I seem to be getting quite different files:


$ GZIP="--rsyncable" tar zcf m5.tgz ./newdir
$ GZIP="--rsyncable" tar zcf m6.tgz ./newdir
$ sum -r m[56]*
25302  2929 m5.tgz
60054  2929 m6.tgz
$ cmp m[56]*
m5.tgz m6.tgz differ: byte 5, line 1



I also tried this with the other format:


$ tar cf - ./newdir | gzip --rsyncable > m7.tgz
$ tar cf - ./newdir | gzip --rsyncable > m8.tgz
$ sum -r m[78]*
36275  2929 m7.tgz
15148  2929 m8.tgz
$ cmp m[78]*
m7.tgz m8.tgz differ: byte 5, line 1


I tried this:

$ tar cf m9.tar ./newdir
$ tar cf m10.tar ./newdir
$ sum -r m*tar
54744  3340 m10.tar
54744  3340 m9.tar
$ gzip --rsyncable m9.tar
$ gzip --rsyncable m10.tar

$ sum -r m*tar
21783  2929 m9.tar.gz
30175  2929 m10.tar.gz


It seems that gzip produces different output even given the same input. Not very rsync friendly.
I did these tests on Debian Lenny v5.0.3 with gzip v1.3.12.

Have I missed the point?




I'm not sure I can see what I'd use the rdiff command for since rsync will do the delta-transfer cleverness for me.

Cheers,
Mark


> Subject: RE: rsync of STDIN to a file.
> From: matt at mattmccutchen.net
> To: mark_young at hotmail.com
> CC: rsync at lists.samba.org
> Date: Thu, 19 Nov 2009 11:53:30 -0500
> 
> On Thu, 2009-11-19 at 16:28 +0000, Mark Young wrote:
> > I tried an experiment to see how rsync coped with the tar compressed
> > files versus the uncompressed files. I took a .tgz from last week and
> > rsync'd it with last nights version. rsync achieved the transfer with
> > a 1.61 speed up. Specifically 172MB of the 262MB file was transferred.
> > I then tried the same thing with the uncompressed tar files. rsync
> > achieved the transfer with a 690 speed up. Specifically 5MB for the
> > 3.6GB file was transferred. Clearly if diskspace were not an issue
> > this would be by far the superior option. Obviously I could increase
> > the complexity of the overall solution with compress & uncompress jobs
> > in sync with the backup strategy, but I believe in keeping it as
> > simple as possible when it comes to backups.
> 
> Right.  The gzip --rsyncable option is designed for this case: files
> compressed with --rsyncable should delta-transfer almost as well as
> uncompressed files, at a minor penalty in the compression ratio.
> 
> > rdiff is a new one on me. Very interesting and something I'll
> > experiment with. For our strategy of 7 daily backups and 12 monthly
> > backups I guess I'd need to use two rdiff destinations so I could
> > apply the --remove-older-than 1W and --remove-older-than 365D options
> > respectively. It might not suit my current situation as explained
> > above regarding storing a tarball versus a directory hierachy, but
> > it's definitely worth knowing about.
> 
> I was talking about rdiff (a command-line interface to the
> delta-transfer algorithm), not rdiff-backup (the incremental backup
> tool).
> 
> > Also I'd not heard of the --rsyncable option for tar. It's not there
> > on my Debian Etch system, nor in my cygwin tar. So I guess it's not in
> > common distributions by default. Still it sounds useful and is
> > something I will bear in mind as something to consider.
> 
> The option applies to gzip, not tar.  In fact, it looks like Fedora
> applies an updated version of the gzip-rsyncable patch to their gzip
> package.  You can get their patch from:
> 
> http://cvs.fedoraproject.org/viewvc/rpms/gzip/F-11/gzip-1.3.9-rsync.patch?view=co
> 
> -- 
> Matt
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20091119/f7e8ccb4/attachment.html>


More information about the rsync mailing list