Rsync died on big directories/file-lists

King, Daniel Daniel.King at Fiserv.com
Wed Dec 10 06:25:01 EST 2003


Vu,

I see the code 12 exit quite frequently over out WAN (using ssh as the
transport).  It is generally a network hiccup.  What I ended up doing was
including rsync in a shell script which retries if the return code is not 0,
i.e.:

#!/usr/bin/ksh

count=0
while :
do
	((count=count+1))

	if [ $count -gt 5 ]
	then
		echo "TOO MANY RETRIES #$count"
		break
	fi

	rsync $your $options $here
	rc=$?
	if [ $rc -ne 0 ]
	then
		echo "RETRY #$count (rsync FAILED with rc=$rc)"
	else
		break
	fi

done

Unfortunately for large directories, this does mean that you reprocess
everything.  I've rarely had more than one retry unless there was a serious
network outage.

A. Daniel King, System Analyst
HP-UX, Solaris, Linux

-----Original Message-----
From: jw schultz [mailto:jw at pegasys.ws] 
Sent: Tuesday, December 09, 2003 2:07 PM
To: rsync at lists.samba.org
Subject: Re: Rsync died on big directories/file-lists

On Tue, Dec 09, 2003 at 01:54:25PM -0500, Vu Huynh wrote:
> 
> 
>   Hi,
> 
>      My name is Vu Huynh.  I'm currently work for Atmel Corporation.
We're 
>      using Rsync v2.5.7 to rsync design kits between our remote sites.  It
works 
> fine for all small kit directories.  However, there is 1 BIG kit directory

> (But there is no file has the size of bigger than 100MB) that always gets 
> "rsync: connection unexpectedly closed" Error.  The Full Error Message is 
> below:
> 
>  % rsync --password-file  -avz user at host::KIT1 KIT1 --exclude '5.*' 
>  --exclude '6.*'
> receiving file list ... done
> ./
> 7.0/
> 7.0/bin/
> 7.0/bin/AWK/
> 7.0/bin/IA32,Linux_2.4.7-10,32bits/
>     ..............
>     ..............
> 7.0/tools_sources/
> 7.0/tools_sources/templates/
> rsync: connection unexpectedly closed (1892713 bytes read so far)
> rsync error: error in rsync protocol data stream (code 12) at io.c(165)
> 
>     Does file-list has a size limitation?  Or something was not setup
right 
>     in this kit directory? I hope you can have any clue/suggestion of what
was 
>  wrong or limitation of rsync.

If you have gotten to the point of processing files it is
unlikely that the size of the file list is the issue.

With the information given my best guess is that something
is causing a delay where no data is being sent and the
firewall is dropping the session.

You might try another -v option to see if you can get more
specific info about what file rsync is processing or other
possible causes for a stall.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt
-- 
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



More information about the rsync mailing list