DO NOT REPLY [Bug 4162] New: Wanted: a mechanism to prevent rsync network compression of compressed files

[samba-bugs at samba.org]

https://bugzilla.samba.org/show_bug.cgi?id=4162

           Summary: Wanted: a mechanism to prevent rsync network compression
                    of compressed files
           Product: rsync
           Version: 3.0.0
          Platform: PPC
        OS/Version: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: rsync at name99.org
         QAContact: rsync-qa at samba.org


rsync, of course, offers the -z flag to allow for the transfer of compressed
file over the network during a backup. 
My experience using this flag has been that (on a 1GHz PPC laptop connected
using 802.11g)
- without using the -z flag the average data rate over the connection is about
2.5MB/s which is about where you'd expect 802.11g to max out, all things
considered
- with using the -z flag the average data rate over the connection is about
1MB/s, and the CPU is maxed out.

Now, if what was being transferred was a stream of text (compression ratio of
say 4 or so), this would still be a win. But on a modern personal system, the
bulk of the material transferred (by bytes, not by file number) is going to be
photos, audio files and video files, ie already compressed stuff, so the mean
compression rate over the entire stream of data is going to be just a bit over
1, and using -z is a loss.

The obvious issue, then, is how can we get the goodness of -z for text files,
while avoiding the cost of the CPU to compress files that aren't going to
compress much. 

Two obvious strategies spring to mind:
* We could track the progress of the compression and bail out if it is less
than some lower limit, maybe 1.2 or so. Maybe run compression till about 8KiB
into the file, see how things are going and coontinue compressed or switch to
uncompressed. AND/OR
* We could simply allow for a user-supplied list of files, (presumably the same
syntax as backup_excludes) that we would not bother to try to compress. This
may not be as robust a strategy as the first scheme, but it is much easier to
program, and should be good enough for most purposes. I'd recommend in addition
that rsync ship with a starter template for this file that includes all the
usual suspects from *.gzip through *.mp3, *.mov, *ogg etc etc.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.