[Bug 3117] New: rsyncing a file to a partially downloaded copy is extremely slow

[samba-bugs at samba.org]

https://bugzilla.samba.org/show_bug.cgi?id=3117

           Summary: rsyncing a file to a partially downloaded copy is
                    extremely slow
           Product: rsync
           Version: 2.6.6
          Platform: All
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: prumpf at gmail.com
         QAContact: rsync-qa at samba.org


I would imagine that it is common to use rsync to rsync a file to a partial
download of itself (or a prefix of the file that might arise from something
being appended to the source file).  However, for large files, this seems to be
extremely slow since many small chunks of constant size are compared.

While the --block-size option can help with this, the block size to use has to
be calculated for each rsync invocation to avoid retransmitting an average of
block-size/2 bytes.

I would suggest that modifying the rsync algorithm to initially compare chunks
of exponentially increasing size until a mismatch is found would probably be
worth it in terms of the total bandwidth saved.  Even if you disagree with that,
a quick-and-dirty fix would be an option that would cause rsync to check for the
case that the larger file results from the smaller file by appending data before
going into the full rsync algorithm.  I believe this wouldn't take more than a
couple of minutes for someone familiar with rsync internals.

It certainly seems odd that rsync is essentially unusable for something that
wget --continue deals with.

To help searching for this bug: log files append appending live streams partial
download aborted download interrupted download restarting rsync restart

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.