AIX 5.1 rsync large file

[Jackie.E.Andrason at Quintiles.com]

Thank you for your response.

I compiled rsync 2.6.7 and installed in and that did the trick.  I don't
know if it had the dynamic_hash patch or not.  But I think that I was too
impatient previously and the 2.6.4 would have worked had I not killed it.

thanks!

Jackie Andrason
Sr. Global IT Infrastructure Analyst
Office:  919-998-2933
Fax:      919-998-2339
Jackie.Andrason at quintiles.com


                                                                           
             "Shachar Shemesh"                                             
             <rsync at shemesh.bi                                             
             z>                                                         To 
                                       Jackie.E.Andrason at Quintiles.com     
             03/24/2006 12:14                                           cc 
             AM                        rsync at lists.samba.org               
                                                                   Subject 
                                       Re: AIX 5.1 rsync large file        
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




Let's try.


Jackie.E.Andrason at Quintiles.com wrote:

>There is one file that is over 45 GB, and it started having
>trouble with that file.
>
>  I don't know what the size of the file was before
>it started having trouble
>
>We tried pushing and pulling both, and it gets that error both ways.
>
Rsync has two, unrelated, definitions for connection endpoints. The side
initiating the connection is called the "client", while the other side
is called the "server". Then again, the side that has the original files
to be synched is called the "sender", while the other side is called the
"receiver".

Switching from pushing to pulling switches the client and server sides,
but does nothing to change the sender and receiver tags. The code that
freaks out on you is in the sender, and so the switch has no effect.

>The error message was:
>ERROR: out of memory in build_hash_table
>
>ulimit -a shows:
>memory(kbytes)       32768
>
>
So you are telling rsync to synchronize a 45GB file using no more than
32MB of memory? Try setting it much higher, if "unlimited" is not an
option. Under "bash" and Linux, this is done with "ulimit -v unlimited".
I'm not sure that the corresponding AIX command will be the same.

>Any help and suggestions would be appreciated, I'm at my wit's end with
>this stuff, since I've never used it previous to this.
>
>
Yes.

rsync 2.6.7 has a change that is meant to reduce the memory consumption
of the hash table during transfer of large files. While on the subject,
the "patches" directory has a patch that is meant to speed things up
when using files larger than 2.5GB. Switching to 2.6.7 is, therefor,
adviseable (though it may well be that 32MB of memory will still not be
enough).

While on the matter, any feedback regarding comparing 2.6.7 with and
without the "dynamic_hash" patch would be greatly appreciated. I'm
trying to push Wayne into putting it into the actual rsync code. A
benchmark saying "it took so and so time with vanilla 2.6.7 and so and
so with 2.6.7 + dynamic_hash" would be much appreciated.

          Shachar


****************IMPORTANT--PLEASE READ*******************
This electronic message, including its attachments, is COMPANY CONFIDENTIAL
and may contain PROPRIETARY or LEGALLY PRIVILEGED information.  If you are 
not the intended recipient, you are hereby notified that any use, disclosure,
copying, or distribution of this message or any of the information included
in it is unauthorized and strictly prohibited.  If you have received this
message in error, please immediately notify the sender by reply e-mail and
permanently delete this message and its attachments, along with any copies
thereof. If this electronic message contains a zipped attachment and you do
not have a decompression tool, you may download unZIP (free of cost) from:
http://www.mk-net-work.com/us/uz/unzip.htm. Alternatively, you may request
that the attachment be resent in an uncompressed format.        Thank you. 
************************************************************************