<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Ok thanks Matt,<br><br>Ah. I think I've realised my mistake. I saw this webpage <a href="http://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/">http://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/</a> which showed the following command.<br> GZIP="--rsyncable" tar zcvf toto.tar.gz /toto<br><br>I've just realised that gzip must be picking up the environment variable when it's invoked by tar. Cool.<br><br>So if I understand you correctly you are suggesting that I modify my backup line to be something like this:<br><br>tar czf - --files-from $FILE_LIST | gzip --rsyncable > ${BACKUP_FILE}<br>or:<br>GZIP="--rsyncable" tar cvf - --files-from $FILE_LIST > ${BACKUP_FILE}<br>
<br>then<br>rsync ${BACKUP_FILE} $HOST:${BACKUP_FILE}<br><br>I see that Debian's gzip has --rsycnable available. Hoorah.<br><br>I just tried some tests but didn't get the results I expected. I seem to be getting quite different files:<br><br><br>$ GZIP="--rsyncable" tar zcf m5.tgz ./newdir<br>$ GZIP="--rsyncable" tar zcf m6.tgz ./newdir<br>$ sum -r m[56]*<br>25302 2929 m5.tgz<br>60054 2929 m6.tgz<br>$ cmp m[56]*<br>m5.tgz m6.tgz differ: byte 5, line 1<br><br><br>
I also tried this with the other format:<br>
<br>$ tar cf - ./newdir | gzip --rsyncable > m7.tgz<br>$ tar cf - ./newdir | gzip --rsyncable > m8.tgz<br>$ sum -r m[78]*<br>36275 2929 m7.tgz<br>15148 2929 m8.tgz<br>$ cmp m[78]*<br>m7.tgz m8.tgz differ: byte 5, line 1<br><br><br>I tried this:<br><br>$ tar cf m9.tar ./newdir<br>$ tar cf m10.tar ./newdir<br>$ sum -r m*tar<br>54744 3340 m10.tar<br>54744 3340 m9.tar<br>$ gzip --rsyncable m9.tar<br>$ gzip --rsyncable m10.tar<br>
$ sum -r m*tar<br>21783 2929 m9.tar.gz<br>30175 2929 m10.tar.gz<br><br><br>It seems that gzip produces different output even given the same input. Not very rsync friendly.<br>I did these tests on Debian Lenny v5.0.3 with gzip v1.3.12.<br><br>Have I missed the point?<br><br><br><br><br>I'm not sure I can see what I'd use the rdiff command for since rsync will do the delta-transfer cleverness for me.<br><br>Cheers,<br>Mark<br><br><br>> Subject: RE: rsync of STDIN to a file.<br>> From: matt@mattmccutchen.net<br>> To: mark_young@hotmail.com<br>> CC: rsync@lists.samba.org<br>> Date: Thu, 19 Nov 2009 11:53:30 -0500<br>> <br>> On Thu, 2009-11-19 at 16:28 +0000, Mark Young wrote:<br>> > I tried an experiment to see how rsync coped with the tar compressed<br>> > files versus the uncompressed files. I took a .tgz from last week and<br>> > rsync'd it with last nights version. rsync achieved the transfer with<br>> > a 1.61 speed up. Specifically 172MB of the 262MB file was transferred.<br>> > I then tried the same thing with the uncompressed tar files. rsync<br>> > achieved the transfer with a 690 speed up. Specifically 5MB for the<br>> > 3.6GB file was transferred. Clearly if diskspace were not an issue<br>> > this would be by far the superior option. Obviously I could increase<br>> > the complexity of the overall solution with compress & uncompress jobs<br>> > in sync with the backup strategy, but I believe in keeping it as<br>> > simple as possible when it comes to backups.<br>> <br>> Right. The gzip --rsyncable option is designed for this case: files<br>> compressed with --rsyncable should delta-transfer almost as well as<br>> uncompressed files, at a minor penalty in the compression ratio.<br>> <br>> > rdiff is a new one on me. Very interesting and something I'll<br>> > experiment with. For our strategy of 7 daily backups and 12 monthly<br>> > backups I guess I'd need to use two rdiff destinations so I could<br>> > apply the --remove-older-than 1W and --remove-older-than 365D options<br>> > respectively. It might not suit my current situation as explained<br>> > above regarding storing a tarball versus a directory hierachy, but<br>> > it's definitely worth knowing about.<br>> <br>> I was talking about rdiff (a command-line interface to the<br>> delta-transfer algorithm), not rdiff-backup (the incremental backup<br>> tool).<br>> <br>> > Also I'd not heard of the --rsyncable option for tar. It's not there<br>> > on my Debian Etch system, nor in my cygwin tar. So I guess it's not in<br>> > common distributions by default. Still it sounds useful and is<br>> > something I will bear in mind as something to consider.<br>> <br>> The option applies to gzip, not tar. In fact, it looks like Fedora<br>> applies an updated version of the gzip-rsyncable patch to their gzip<br>> package. You can get their patch from:<br>> <br>> http://cvs.fedoraproject.org/viewvc/rpms/gzip/F-11/gzip-1.3.9-rsync.patch?view=co<br>> <br>> -- <br>> Matt<br>> <br> </body>
</html>