AIX 5.1 rsync large file

[Shachar Shemesh]

Let's try.


Jackie.E.Andrason at Quintiles.com wrote:

>There is one file that is over 45 GB, and it started having
>trouble with that file.
>
>  I don't know what the size of the file was before
>it started having trouble
>
>We tried pushing and pulling both, and it gets that error both ways. 
>
Rsync has two, unrelated, definitions for connection endpoints. The side
initiating the connection is called the "client", while the other side
is called the "server". Then again, the side that has the original files
to be synched is called the "sender", while the other side is called the
"receiver".

Switching from pushing to pulling switches the client and server sides,
but does nothing to change the sender and receiver tags. The code that
freaks out on you is in the sender, and so the switch has no effect.

>The error message was:
>ERROR: out of memory in build_hash_table
>
>ulimit -a shows:
>memory(kbytes)       32768
>  
>
So you are telling rsync to synchronize a 45GB file using no more than
32MB of memory? Try setting it much higher, if "unlimited" is not an
option. Under "bash" and Linux, this is done with "ulimit -v unlimited".
I'm not sure that the corresponding AIX command will be the same.

>Any help and suggestions would be appreciated, I'm at my wit's end with
>this stuff, since I've never used it previous to this.
>  
>
Yes.

rsync 2.6.7 has a change that is meant to reduce the memory consumption
of the hash table during transfer of large files. While on the subject,
the "patches" directory has a patch that is meant to speed things up
when using files larger than 2.5GB. Switching to 2.6.7 is, therefor,
adviseable (though it may well be that 32MB of memory will still not be
enough).

While on the matter, any feedback regarding comparing 2.6.7 with and
without the "dynamic_hash" patch would be greatly appreciated. I'm
trying to push Wayne into putting it into the actual rsync code. A
benchmark saying "it took so and so time with vanilla 2.6.7 and so and
so with 2.6.7 + dynamic_hash" would be much appreciated.

          Shachar