BIG delete list makes for BIGGER RAM requirements

[Clint Byrum]

So I'm doing daily backups with rsync, and weekly, I run it with
--delete after archiving the whole thing (this way I don't lose any
deleted files). All week long this runs fine, but when I add --delete,
rsync runs for a few hours then aborts because the box runs out of
memory.

Jan 30 06:31:09 backup kernel: Out of Memory: Killed process 3490
(rsync).

Source box:
RH8+Custom Patches
Rsync v2.5.7
Linux v2.4.27
RAM: 2GB

Destination box (backup, from above message):
Fedora Core 1
Rsync v2.6.3
Linux 2.6.8
RAM: 512MB
Swap: 256MB
XFS on LVM2 on Software RAID5

Cmdline:

rsync --delete --force --safe-links -a --exclude=log --exclude=cricket
--exclude=backup --exclude=matt --exclude=ftp --exclude=*_tmp*
--exclude=mysql --exclude=lost+found --exclude=db/tempfiles/3hours
--exclude=db/tempfiles/1day --exclude=httpd/backhand* --exclude=jim
--exclude=clint --exclude=*.file --exclude=*tempfile*
--exclude=db/files -e ssh --whole-file
remotebox:/home/ /back/daily/remotebox/home

The box has nothing using up a ton of RAM, so there is always at least
500MB of virtual memory available. I've never been able to capture
exactly how much RAM rsync was using when this happens, but thats
probably about it.

During the week, it transfers about 85,000-100,000 new small files each
run. For --delete, I'm guessing it needs to delete maybe 20 times that
many files sometimes.

Is there a calculation I can do to find out how much RAM I'll need to do
this delete? The files are spread out pretty sporadically, so it would
be a real pain to try and do it in segments. Am I better off just
torching the whole tree on the backup server and retransferring all
60GB? Would it help to upgrade the sending machine to v2.6.3?

Thanks.
-- 
Clint Byrum <cbyrum at spamaps.org>