[Bug 10678] New: performance problem with lots of hard links?

[samba-bugs at samba.org]

https://bugzilla.samba.org/show_bug.cgi?id=10678

           Summary: performance problem with lots of hard links?
           Product: rsync
           Version: 3.0.6
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: Dave at Yost.com
         QAContact: rsync-qa at samba.org


We use rsync to do backups. Each backed up file is hard linked to the
corresponding file from the day before unless the file differs.

On the backup disk there are 3 directory trees, one per backed-up machine, each
with 45 directory trees. The backup disk has about 1TB of backup content. df
says each disk being backed up is about 90GB.

At the moment, we're copying an old backup disk to a new larger backup disk.
It's taking forever and seems to be slowing down. After 8 hours it's copied
only 37%. Maybe this is normal. Don't know.

Thinking that perhaps a later version of rsync would do better, we started
again, overwriting what was already copied. 8 hours later rsync hasn't started
writing any new files yet. CPU time usage is about 1/10 of real time, so rsync
seems I/O bound.

It's possible that rsync is thrashing in the L2 cache, but there is a pretty
big L3 cache on this Xeon chip.

I wonder if you have ever profiled a case like this to see if there is anything
that could be better in the algorithm that manages huge numbers of hard links.

Perhaps yet another option flag could tell rsync that it's working on this kind
of workload, and rsync could throw away bookkeeping data for hard links farther
back than yesterday's backup tree.

F   UID    PID   PPID PRI  NI    VSZ      RSS WCHAN  STAT TTY    TIME COMMAND
4     0 190742 152692  20   0 189568     3728 poll_s S+   pts/5  0:00  |   \_
sudo /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup
4     0 190756 190742  20   0 937948   760856 poll_s S+   pts/5 18:45  |      
\_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup
5     0 190757 190756  20   0 1369872 1202448 sync_b D+   pts/5 19:44  |       
   \_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup
1     0 190758 190757  20   0 792392   605480 poll_s S+   pts/5  0:49  |       
       \_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup

Filesystem     1K-blocks          Used     Available          Use% Mounted_on
/dev/sdb3  1,906,881,372   377,934,500 1,432,082,872           21% /backup
/dev/sdc2  1,442,142,772   986,617,308   382,268,788           73% /oldbackup

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.