AIX 5.1 rsync large file

Jackie.E.Andrason at Quintiles.com Jackie.E.Andrason at Quintiles.com
Thu Mar 23 22:15:17 GMT 2006





Hello,

I have inherited a setup where there are 2 AIX 5.1 systems in 2 separate
sites.  There are large database files that are backed from each site to
the other via rsync.  Currently, it is using rsync version 2.5.4.  It does
it via ssh with the options -avz.  This has all been merrily plodding along
for some time.  There is one file that is over 45 GB, and it started having
trouble with that file.  I don't know what the size of the file was before
it started having trouble or if they made a bunch of changes in this .dbf
fille before it started having trouble.  There are other large files that
are smaller than this (i.e., 30 GB, etc.).   The error message was:
ERROR: out of memory in build_hash_table
rsync error: error allocating core memory buffers (code 22) at util.c(115)
rsync: connection unexpectedly closed (585911608 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
rsync: connection unexpectedly closed (1224 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
Rsyncget completed file transfer at Sat Mar 11 02:49:52 EST 2006

We tried pushing and pulling both, and it gets that error both ways.  I
tried downloading an already compiled version of rsync 2.6.4 (and a version
I compiled as well that includes support for AIX large files), and using
this on both ends, it seems to hang, and the hidden .file.xyz that it
creates stalls at size 2097152.  ulimit -a shows:
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000

Any help and suggestions would be appreciated, I'm at my wit's end with
this stuff, since I've never used it previous to this.

Thanks a lot!

Jackie Andrason
****************IMPORTANT--PLEASE READ*******************
This electronic message, including its attachments, is COMPANY CONFIDENTIAL
and may contain PROPRIETARY or LEGALLY PRIVILEGED information.  If you are 
not the intended recipient, you are hereby notified that any use, disclosure,
copying, or distribution of this message or any of the information included
in it is unauthorized and strictly prohibited.  If you have received this
message in error, please immediately notify the sender by reply e-mail and
permanently delete this message and its attachments, along with any copies
thereof. If this electronic message contains a zipped attachment and you do
not have a decompression tool, you may download unZIP (free of cost) from:
http://www.mk-net-work.com/us/uz/unzip.htm. Alternatively, you may request
that the attachment be resent in an uncompressed format.        Thank you. 
************************************************************************


More information about the rsync mailing list