Rsync stops randomly (using vanished files bash script)

[Samuel Vogel]

I installed the latest CVS Version (today about 1pm). Again, the same 
problem occured...

Andre Majorel schrieb:
> On 2007-08-24 12:02 +0200, Samuel Vogel wrote:
>   
>> Andre Majorel schrieb:
>>
>>     
>>> If you have directories with many files, rsync becomes a memory
>>> hog. I've had it bring down a 1-GB machine while copying a 10-GB
>>> news spool.
>>>       
>> Thanks for the reply. Do you mean on the rsync server or client
>> side?
>>     
>
> I don't recall from which machine the transfer was initiated but I
> think the side that hogs memory is the one where the large
> directory exists. So the first time you run rsync, it's just on
> the source host. On subsequent runs, it's on both hosts.
>
>   
>> I'm sure that there is no file bigger than 500MB. And there
>> can't be any files 1GB, because we have 1GB quotas in place!
>>     
>
> IME, rsync has no problems with large files or even many files.
> What it can't deal with is single directories with many files.
> The algorithm used must require holding in memory some per-file
> data for *all* files in the directory being synced.
>
> If you have a news spool, a big maildir folder or some such
> monstrosity, you might want to try excluding them from the sync
> and see if it makes a difference.
>
> Good luck.
>
>