[Bug 3186] Surprisingly large unshared memory usage

Thu Nov 24 06:07:08 GMT 2005

https://bugzilla.samba.org/show_bug.cgi?id=3186

------- Comment #6 from foner-rsync-bugzilla at media.mit.edu  2005-11-23 23:07 MST -------
I tried adding --delete-during (so the full invocation now looks like "rsync
-vrltH --delete -pgo --stats -z -D --numeric-ids -i --delete-during
--exclude-from=/FS/dirvish/HOST/20051124/exclude
--link-dest=/FS/dirvish/HOST/20051123/tree root at HOST:/
/FS/dirvish/HOST/20051124/tree" and the behavior didn't change.

But now that I think about it, it's not clear if --delete could be a problem in
the first place, because these are dirvish runs.  That means that I'm using -H
and --link-dest to populate a tree that originally starts out empty, and winds
up containing only a very few files that consume actual disk space (whatever
got created or modified since the dirvish run yesterday), and about 2 million
hardlinks into the previous day's tree.  If rsync is writing to an
otherwise-empty tree, it seems to me that --delete has nothing to do---which
makes me wonder why dirvish even bothers to supply it automatically, frankly,
since dirvish -always- starts from an empty destination tree.  Is there some
reason why it makes sense to supply --delete at all?  (Unfortunately, we can't
ask dirvish's original author why he did this, alas.)  Or does --delete cause
process inflation if there -isn't- much to do instead of if there -is-?

Once this run completes in a couple hours (I'm debugging some other, unrelated
things at the same time in this run), I may just blow its tree away and start
over without --delete in any form (by editing the dirvish script) and see if
that changes its behavior, but I'd be pretty mystified if it did unless my
understanding of --delete, --link-dest, and empty destination trees is just
wrong.

Just in case I'm being completely faked out here and the second process really
is sharing most of its memory, here are the top few lines of "top" running on
the destination host:

top - 00:34:30 up 8 days,  8:25,  2 users,  load average: 3.05, 2.00, 0.96
Tasks:  65 total,   3 running,  62 sleeping,   0 stopped,   0 zombie

Cpu(s):  6.0% us, 39.9% sy,  0.0% ni,  0.0% id, 51.3% wa,  2.0% hi,  0.7% si
Mem:    516492k total,   508012k used,     8480k free,    76180k buffers
Swap:  1544184k total,    12476k used,  1531708k free,    68404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10795 root      18   0  259m 253m  676 D 15.2 50.3   0:47.19 rsync
10865 root      16   0  251m 245m  688 S  0.0 48.7   0:00.08 rsync

What's actually kinda interesting there is that it claims to have 8m free, and
76m of buffers, -and- to have 253+245m of rsync resident, all on a machine with
only 512m total memory (and not including ~30m of other processes).  And yet
I'm pretty sure I -saw- the free memory go from about 200m to about 0 when that
second process started up on, on previous runs.  (On this one, I didn't quite
catch it in the act and am not sure how much free memory there was before the
inflation of the second process.)

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.