BIG delete list makes for BIGGER RAM requirements

Clint Byrum cbyrum at spamaps.org
Sun Feb 6 18:50:53 GMT 2005


John Van Essen wrote:

>On Mon, 31 Jan 2005, Clint Byrum <cbyrum at spamaps.org> wrote:
>  
>
>>So I'm doing daily backups with rsync, and weekly, I run it with
>>--delete after archiving the whole thing (this way I don't lose any
>>deleted files). All week long this runs fine, but when I add --delete,
>>rsync runs for a few hours then aborts because the box runs out of
>>memory.
>>    
>>
>
>When a --delete option is given, the receiver not only has in memory
>the list of files from the sender, but it also builds its own list of
>local files so it can determine what to delete.  This means that the
>memory requirements are roughly doubled when you do deletions.
>
>However - good news!
>
>Wayne has recently implemented a --delete-during option that deletes
>files in directories as they are processed.  This means there's no more
>monolithic filelist for the local files on the target.
>
>Wayne - it appears that the monolithic target file list is no longer
>built for --delete or --delete-after as well.  Is that true?  If so,
>that should be mentioned in the NEWS, as it will be an improvement
>over the current procedure.
>
>Clint - try the current version in CVS (or use a nightly snapshot).
>It might solve your out-of-memory problem.  Use with --dry-run, first,
>to see how things go.
>  
>
Looks good! --delete-during worked flawlessly with dry-run. Its still 
running "live" right now, but seems to be working perfectly. Memory 
consumed was just 82M. :-D

When is 2.6.4 going to be released?


More information about the rsync mailing list