Rsync dies

[Allen, John L.]

In my humble opinion, this problem with rsync growing a huge memory
footprint when large numbers of files are involved should be #1 on
the list of things to fix.  It seems that every fifth post is a
complaint about this problem!  Sorry if this sounds like ungrateful
whining, but I think rsync would be greatly improved if the overall
way it processes multiple files were overhauled to be much more
resource friendly.

John. 

-----Original Message-----
From: tim.conway at philips.com [mailto:tim.conway at philips.com]
Sent: Friday, May 17, 2002 01:44 PM
To: C.Zimmermann
Cc: rsync at lists.samba.org; rsync-admin at lists.samba.org
Subject: Re: Rsync dies


Yeah.  You'll have to find a way to break the job up into smaller pieces. 
It's a pain, but I have a similar situation - 3M+ files in 130+Gb.  I 
can't get the whole thing in one chunk, no matter how fast a server with 
however much memory, even on Gb ethernet (for the server).  In my case, 
the filesystem is on NAS, and the NAS has only 100bT simplex (half-duplex, 
to some).
I have some code that can be used to analyze your system before the sync, 
and choose directories containing no more than a maximum number of items 
below them.  Iterating through the list and using -R can let you get the 
whole thing run, though --delete and -H become less certain (not 
dangerous, but if you don't name anything containing a deleted directory 
because it didn't come up on your list, youll never tell the destination 
to delete it, and if you have two hard links to the same file, but hit 
them in two seperate runs, you now have two copies on disk).
Let me know if you want it.  I'm sure you can figure how to modify it for 
your environment.

Tim Conway
tim.conway at philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn, 
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), 
".\n" '
"There are some who call me.... Tim?"




"C.Zimmermann" <clemens at prz.tu-berlin.de>
Sent by: rsync-admin at lists.samba.org
05/17/2002 02:08 AM

 
        To:     <rsync at lists.samba.org>
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        Rsync dies
        Classification: 



I´m trying to rsync a 210 GB Filesystem with approx 1.500.000 Files.

Rsync always dies after about 29 GB without any error messages.
I´m Using rsync  version 2.5.5  protocol version 26.

Has anyone an idea ?

Thank´s Clemens



-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html




-- 
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html