Long pause.

[jw schultz]

On Tue, Aug 26, 2003 at 12:45:10PM +0200, Rogier Wolff wrote:
> 
> Hi everyone, 
> 
> I'm trying to rsync a large directory. After pondering on the 
> list-of-files, it's now more or less doing nothing. It's
> running for over 4 hours now, not doing anything except
> eating CPUtime on the destination for over half an hour. I don't
> know what it was doing before that. 
> 
> The stdout of the rsync reported: 
>     4675350 files to consider
> 
> (which is about right)
> 
> On the source machine it's using about 470 Mb of memory.
> On the destination machine it's using 800M of memory and growing.
> The rsync process on the destination machine  is not doing
> ANY system calls. Oh! There ARE a lot of hardlinks involved. 

It cannot grow without doing system calls.

470MB would be consistent with the file list size you site.
Another 300MB+ additional could be due to a large file but
i'd have to double check source which i'm not going to do at
this moment.  Some of that might be due to the hardlinks but
again i'd have to check the source to see how much.

> The destination machine IS swapping a bit:
> 
[snip]

Swap may be thrashing.

> Is this normal?
> 
> Is there a way that I can estimate how much longer this is going to
> take.

No.

> (A day: fine. A week: Not ok.) Will rsync use MORE memory? I
> have 200m of swap free right now, but if I should expect it to double
> its memory use once more, I will have to add some swap to make it 

probably not by much.  But who can tell.

> happy..... Right now I can monitor it, but I'll need some sleep
> in the next 16 hours or so... ;-)

You don't say what version of rsync, what OS, or what
options were specified.

One thing is certain. Reducing the file count per invocation
will reduce the memory footprint although it won't be able
to preserve hardlinks that cross invocation boundaries.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt