[Bug 12570] New: Problems with --checksum --existing

samba-bugs at samba.org samba-bugs at samba.org
Tue Feb 7 12:19:08 UTC 2017


            Bug ID: 12570
           Summary: Problems with --checksum --existing
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: atom at smasher.org
        QA Contact: rsync-qa at samba.org


I've got an sd-card with some movies, a few of which are corrupted files.

I want to copy only the files that don't match the good files.

 rsync --checksum --existing -vhriP /movies/ /media/128-SD/Movies/

The problem here is that *all* files in "/movies/" are hashed before anything
else happens. This can be verified with lsof: "lsof +D /movies".

I've got <100GB in "/media/128-SD/Movies/".

I've got >1.5TB in "/movies/", and hashing all of those files is just a huge
waste of time and system resources.

When "--existing" and "--checksum" are both used, the algorithm should first
make a list of candidate files, then start hashing. It should *not* start
hashing everything on the send-side and then figure out which files might be

Workaround for me:
 diff -r /movies/ /media/128-SD/Movies/ | grep differ | awk '{print "pv " $3" >
"$5}' | sh

nb, that workaround requires "pv" and only works with file-names that do not
contain spaces, but for me it's a quick and easy way to see progress while
files are being copied. "cp" would work fine in place of "pv".

On my system, that workaround saved my about 1-2 days of hashing, and completed
in less than an hour.

You are receiving this mail because:
You are the QA Contact for the bug.

More information about the rsync mailing list