TODO hardlink performance optimizations

John Van Essen vanes002 at
Sun Jan 4 12:35:03 GMT 2004


You articulated your situation clear enough for me.  Thanks.

I'll address your issue about when rsync is running locally for /vol/N
to /vol/N_mirror syncing, it exhausts all of the RAM and swap.

If you haven't read jw schultz's "How Rsync Works" page, here the link:

The sender, receiver, and generator each have a full copy of the file
list (each file's entry uses 100 bytes on average).

Additonally, the --hard-links option creates yet *another* full copy of
the file list in the receiver, so that's even more memory consumed.

So you are in a world o' hurt rsyncing an entire /vol/N internally
with --hard-links, since there will be FOUR copies of the file list.

I'd suggest breaking the /vol/N rsync up into separate rsyncs for each
of the maxdepth 1 hierarchies.  If I understand your situation correctly,
all hard link groups are self contained within each of those hierarchies
so you will be OK.

I've modified hlink.c to use a list of file struct pointers instead of
copies of the actual file structs themselves, so that will save memory.
I'll submit that patch for review in a day or two after I've tested it.
        John Van Essen  Univ of MN Alumnus  <vanes002 at>

On Sat, 3 Jan 2004, Lester Hightower <hightowe-rsync-list at> wrote:
> Hello,
> I read with interest the mailing list thread found here:
> We have a "situation" with rsync and --hard-links that was the reason for
> my search in MARC's rsync list archive that turned up the thread shown
> above.  After reading through that thread, and other information on this
> topic, I believe that sharing our situation with you will in itself prove
> to be a good contribution to rsync (which is an excellent tool, BTW).
> So, here goes:
> We have a process on a backup server (I called it "s" below), that each
> night rsyncs a full copy of /, /var, and /usr from a great number of
> systems.  As a rule we put /, /var, and /usr on separate partitions, but
> that detail is not important.  What is important is to understand exactly
> how we do these nightly, full system backups.  First, let me start by
> showing you what a small set of the system_backups hierarchy looks like:
> root at s:/vol/6/system_backups# find . -type d -maxdepth 1
> .
> ./client1
> ./docs1.colo1
> ./docs2.colo1
> ./ipfw-internal.colo1
> ./ipfw1
> ./ipfw2
> ./docsdev1
> root at s:/vol/6/system_backups# find . -type d -maxdepth 2|head -25|egrep -v '^\./[^/]+$'|sort
> .
> ./client1/20031223
> ./client1/20031224
> ./client1/20031225
> ./client1/20031226
> ./client1/20031227
> ./client1/20031229
> ./client1/20040102
> ./client1/current
> ./docs1.colo1/20031219
> ./docs1.colo1/20031223
> ./docs1.colo1/20031224
> ./docs1.colo1/20031225
> ./docs1.colo1/20031226
> ./docs1.colo1/20031227
> ./docs1.colo1/20031229
> ./docs1.colo1/20040102
> ./docs1.colo1/current
> ./docs1.colo1/image-20031218
> ./docs2.colo1/20031218
> ./docs2.colo1/20031219
> ./docs2.colo1/current
> OK, that gives you an idea of how the hierarchy looks.  Here is the critical
> part, though.  The logic that creates these each night looks like this:
> TODAY=<YYYYMMDD for today>
> for HOST in (<hosts>); do
>   cp -al $HOST/current $HOST/$TODAY
> rsync remote $HOST into my local $HOST/current...
> done
> For those not familiar with the -l option to cp:
> root at s:/vol/6/system_backups# man cp|grep -B1 -A1 'hard links instead'
>        -l, --link
>               Make hard links instead of copies  of  non-directo-
>               ries.
> What we end up with is a tree that is _very_ fast to rsync each night,
> with revision history going back indefinitely, at the disk usage cost of
> only files that change (rare) and the directories (about 8MB per machine).
> Note, however, that the _vast_ majority of file entries on these file
> systems (system_backups) are hard links.  Many inodes will have 20, 30, or
> more filename entries pointing at them (depending strictly on how much
> history we choose to keep).
> Keeping all that in mind, now understand that server "s" has /vol/(0..14)
> installed in its disk subsystem, and (the important part) each of those
> volumes has a slow mirror -- one rsync per day.  We do not keep those
> mirrors mounted, but you could think of /vol/0 having a /vol/0_mirror
> partner that is rsynced once every twenty-four hours.
> All of this works absolutely perfectly, with one exception, the daily
> rsync of /vol/N to /vol/N_mirror for volumes that hold system_backups, and
> the reason appears to be the --hard-links flag.  Rsync, which is running
> completely locally for /vol/N to /vol/N_mirror work, exhausts all of the
> RAM and swap allocated to it in this machine (3GB), sends the machine into
> a maddening swap spiral, etc.  The issue only exists for /vol/N vols where
> we have "system_backups" stored.
> I wanted to share this circumstance with you because my reading of the
> discussion on this topic, though encouraging, left me with the impression
> that some might not be thinking about situations like this one, where it
> is perfectly normal and desired to have many hard links to one inode, and
> hundreds of thousands of hard links in one file system.
> To give you an idea of the type of information one can glean from such a
> backup process, here are a couple of examples.  Keep in mind that files
> with link-count of 1 changed on the date indicated by the directory:
> root at s:/vol/6/system_backups/client1# find 20040102 -links 1 -type f|head -2
> 20040102/root/.bash_history
> 20040102/tmp/.803.e4a1
> root at s:/vol/6/system_backups/client1# diff 20040102/root/.bash_history current/root/.bash_history
> 1d0
> < lynx http://localhost:1081 --source | grep Rebuilding | head -1 | cut 10-
> 500a500
>> ssh ljacobs at supermag
> root at s:/vol/6/system_backups/client1# find 20040102 -links 1 -type f|cut -d/ -f1,2,3,4|sort |uniq -c
>       1 20040102/SYMLINKS
>       1 20040102/root/.bash_history
>       1 20040102/tmp/.803.e4a1
>       1 20040102/usr/local/BMS
>      54 20040102/usr/local/WWW
>      17 20040102/usr/local/etc
>       1 20040102/usr/sbin/symlinks
>      42 20040102/vol/1/bmshome
>       1 20040102/vol/2/webalizer_working
>      12 20040102/vol/3/home
> You'll notice that the hard link counts in this file system are not very
> high yet (only 8), yet it is _very_ intensive to have rsync try to sync
> /vol/6system_backups/client1 to /vol/6_mirror/system_backups/client1 with
> the --hard-links flag set:
> root at s:/vol/6/system_backups/client1# find 20040102 ! -links 1 -type f -printf '%n\t%i\t%s\t%d\t%h/%f\n'|head -50|tail -5
> 8       11323   10108   2       20040102/bin/mknod
> 8       11324   25108   2       20040102/bin/more
> 8       11325   60912   2       20040102/bin/mount
> 8       11326   10556   2       20040102/bin/mt-GNU
> 8       11327   33848   2       20040102/bin/mv
> If there is anything that I did not articulate clearly, if you have any
> followup questions, if you would like us to test some code for you guys,
> or if there is anything else that you feel that I can do to help, please
> do not hesitate to ask.
> Sincerely,
> --
> Lester Hightower
> 10East Corp.
> p.s.  10East created and now supports the MARC system ( in
> various ways, including hosting it, though it is primarily administered by
> Mr. Hank Leininger, a good friend and former employee.  I didn't see any
> mention of MARC in the rsync web-site.  Please feel free to use it.

More information about the rsync mailing list