Server crashed using rsync

[jw schultz]

On Wed, Jan 28, 2004 at 02:12:02PM -0500, Eugen Luca wrote:
> I'm trying to make a backup using this command 
>  
>     rsync -auvH /home/ /bak --delete --bwlimit=1000 --status
>  
> server load has been increased so much and the server crashed, as well
> has gone out of memory 
> My Server is a Dual Xeon 2.0 GHz with 2GB of Memory + 1GB Swap.
> Could be that there are too many files, about 5.000.000, to be backed up

The file list size is part of your problem particularly for
local transfer.  At approximately 100 bytes per file
you are near the limits of per processs allocatable space.
Add 72 bytes per file (non CVS HEAD) for the -H on the generator
and it gets worse.  Now factor in the fact that for local
transfer the file list (but not hlink list) is built twice
(not shared) and you are pushing/exceeding 1GB RSS just for
rsync.

If this causes the OS to crash you have an unrelated problem
that rsync i just revealing.

> ?
> The way the files are structured make very difficult to create several
> backup with less files.
> Is there any one else facing issues like this ? Is there any workaround
> on this ?

Many people have faced this issue.  I'd suggest perusing the
archives.  There are enhancements in CVS that reduce the
increased memory requirements of -H.

How it may be worked around will depend on your fileset.
You have already indicated that your structure makes
breaking up job difficult.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt