rsync freezes when copying several million files
Tomasz Chmielewski
tch at syneticon.net
Mon Aug 21 13:32:35 GMT 2006
I just tried to copy several million files with rsync.
Unfortunately, it seems to freeze:
# rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ /mnt/iscsi_backup/backuppc/
building file list ...
7891370 files to consider
And it waits here. It was already copied once sussessfully, and this time, I just wanted to do an update.
Destination filesystem is iSCSI, on another device in LAN.
I can list the files there, create new files etc., so it's not a device problem.
As there were many files, it used up lost of RAM:
# free
total used free shared buffers cached
Mem: 524460 520928 3532 0 344 6604
-/+ buffers/cache: 513980 10480
Swap: 1534072 1100884 433188
But still, there is a lot of swap left.
I can see that rsync is in "D" state, and doesn't use much CPU time:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23311 root 5 -17 685m 440m 488 D 0.0 86.0 2:05.94 rsync
23310 root 0 -17 578m 608 608 S 0.0 0.1 8:41.55 rsync
So I made a strace, and rsync wasn't doing much:
[root at syn1 kernel]# strace -p 23311
Process 23311 attached - interrupt to quit
(ctrl + C after one minute)
Process 23311 detached
[root at syn1 kernel]# strace -p 23310
Process 23310 attached - interrupt to quit
select(6, [5], [], NULL, {4, 800000}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}) = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}
(ctrl + C after two minutes or so)
It's in that state for several hours now.
Will it eventually wake up and finish, or perhaps something doesn't work properly?
Those "Timeout" messages in strace output worry me and make me think something didn't go as planned.
I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it looks like that to me).
--
Tomasz Chmielewski
http://wpkg.org
More information about the rsync
mailing list