Has anyone experience with rsync out of memory
mjacob at union06.de
mjacob at union06.de
Fri Aug 24 10:00:01 GMT 2007
Hello,
we are using rsync on AIX and run into out of memory.
There have been some threads about out of memory and FAX
tells that this is a known problem. So I need to finde
a workaround for this.
Please help me!
We try to sync many (nnnn) Thumbnails of size 2k between
AIX and Linux with
rsync -a --delete --rsh=ssh <source dir> <user>@<server>:<target dir>
I calculated 100 bytes a file which lead me to about 350MB
needed for the process. However the process seems to stop
already at getting 16MB.
Is there any restriction to processes on Unix system
that might influence this? Will set ulimit help?
ulimit -a:
-------------------------
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) unlimited
rsync output:
------------------------------
opening connection using ssh <servername deleted> -l was00 rsync --server -vvlogDtpr . /data/thumbnail/0009/
building file list ...
[sender] expand file_list to 131072 bytes, did move
[sender] expand file_list to 262144 bytes, did move
[sender] expand file_list to 524288 bytes, did move
[sender] expand file_list to 1048576 bytes, did move
[sender] expand file_list to 2097152 bytes, did move
[sender] expand file_list to 4194304 bytes, did move
[sender] expand file_list to 8388608 bytes, did move
ERROR: out of memory in receive_file_entry
rsync error: error allocating core memory buffers (code 22) at util.c(116)
------------------------------
vmstat output:
System configuration: lcpu=6 mem=16384MB ent=0.60
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
3 0 4580615 3986 0 5 0 230 860 0 395 7664 2278 60 29 11 0 0.68 113.2
3 1 4580887 4171 0 4 0 115 526 0 320 11402 2586 65 25 10 1 0.94 155.9
2 0 4581595 3988 0 8 0 115 729 0 423 9491 2795 61 24 15 0 1.16 192.9
5 0 4587835 5210 0 8 0 1529 22327 0 290 12330 2584 72 18 9 0 1.29 215.6
1 0 4592992 3958 0 7 0 806 5132 0 264 9062 2775 63 23 14 0 1.25 207.9
2 0 4619807 4479 0 10 892 5475 99037 0 383 10045 3006 52 31 16 0 1.15 191.2
2 1 4623071 3936 0 9 332 579 37071 0 664 7912 3125 57 31 12 1 0.80 133.9
1 1 4623422 4223 0 14 168 237 51655 0 336 9554 2895 63 22 11 3 1.33 221.9
2 0 4623863 4321 0 8 89 115 363 0 286 6629 2316 61 23 15 0 1.45 241.0
3 0 4624786 4191 0 9 83 173 3281 0 397 7922 2631 67 18 14 0 1.48 246.0
3 0 4625560 4544 0 9 50 405 15570 0 493 12824 3510 69 21 11 0 1.54 256.1
2 0 4587483 43290 0 24 161 0 0 0 525 15017 3689 65 24 10 0 1.58 263.8
2 0 4587955 42779 0 5 0 0 0 0 340 11187 3482 34 45 21 0 0.81 134.4
3 0 4588432 42250 0 7 0 0 0 0 303 10849 3025 27 53 20 0 0.67 112.3
2 0 4588967 41603 0 19 0 0 0 0 617 20532 3064 48 34 17 0 0.97 161.1
2 0 4589090 41391 0 13 0 0 0 0 424 9754 2893 48 34 18 0 0.77 128.1
1 1 4589847 37961 0 5 0 0 0 0 427 12108 3557 41 38 20 1 0.92 153.8
1 0 4590468 37293 0 6 0 0 0 0 764 12101 4208 50 34 16 0 1.06 177.2
2 0 4591181 36574 0 3 0 0 0 0 338 10120 3226 61 23 16 0 1.40 234.0
3 0 4591887 35789 0 5 0 0 0 0 385 10640 3410 66 21 13 0 1.48 246.2
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
3 1 4592168 34913 0 23 0 0 0 0 416 10808 3024 68 21 11 1 1.60 266.1
2 0 4592561 34470 0 6 0 0 0 0 492 7663 2362 66 19 15 0 1.56 259.9
1 0 4594123 32863 0 7 0 0 0 0 600 12302 3625 36 39 25 0 0.83 137.6
4 0 4594701 32250 0 3 0 0 0 0 455 12500 3404 50 32 18 0 0.97 161.5
2 0 4595411 31498 0 5 0 0 0 0 370 13597 3569 54 29 16 0 1.11 185.7
2 0 4595919 30925 0 3 0 0 0 0 490 21337 4070 41 42 17 0 0.85 141.1
3 1 4596534 30252 0 9 0 0 0 0 443 10633 3291 60 24 15 0 1.15 191.6
3 0 4597596 29167 0 2 0 0 0 0 488 7748 2476 50 32 18 0 0.82 137.1
3 1 4606000 20730 0 3 0 0 0 0 383 15290 3501 51 30 19 0 1.19 198.6
2 0 4608962 17739 0 3 0 0 0 0 338 10320 3053 41 43 16 0 0.71 118.5
4 0 4636866 4547 0 4 1196 2941 22323 0 538 11462 3689 57 30 13 0 1.24 207.1
1 0 4639108 3957 0 11 195 346 3190 0 614 10910 3858 54 30 16 0 1.16 194.0
1 0 4639912 4000 0 18 115 346 3675 0 361 10948 3302 61 25 14 0 1.54 256.3
2 0 4640020 7240 0 3 475 1035 18606 0 451 13522 3375 69 18 12 0 1.59 265.1
1 0 4640698 5624 0 5 0 0 0 0 529 19685 4387 63 22 14 0 1.71 285.8
2 1 4642203 4052 0 7 0 0 0 0 448 12264 3591 64 22 14 0 1.47 244.6
1 0 4603332 44578 0 3 192 346 4927 0 512 12164 3280 38 34 28 0 0.99 165.8
3 0 4603641 44146 0 3 0 0 0 0 386 12964 3117 37 40 23 0 0.77 128.4
2 0 4603827 43874 0 15 0 0 0 0 326 8267 2048 39 37 24 0 0.65 109.0
2 1 4572145 75451 0 15 0 0 0 0 315 8757 2845 27 48 25 0 0.65 108.2
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 4572202 75369 0 2 0 0 0 0 296 3016 1046 14 19 67 0 0.29 48.8
Thank you for any help.
Kind regards,
Marco
--
Marco Jacob, PMP
Prisma GmbH, Germany
More information about the rsync
mailing list