rsync freezes when copying several million files

[Tomasz Chmielewski]

I just tried to copy several million files with rsync.

Unfortunately, it seems to freeze:

# rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ /mnt/iscsi_backup/backuppc/
building file list ...
7891370 files to consider


And it waits here. It was already copied once sussessfully, and this time, I just wanted to do an update.
Destination filesystem is iSCSI, on another device in LAN.
I can list the files there, create new files etc., so it's not a device problem.


As there were many files, it used up lost of RAM:

# free
             total       used       free     shared    buffers     cached
Mem:        524460     520928       3532          0        344       6604
-/+ buffers/cache:     513980      10480
Swap:      1534072    1100884     433188

But still, there is a lot of swap left.


I can see that rsync is in "D" state, and doesn't use much CPU time:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23311 root       5 -17  685m 440m  488 D  0.0 86.0   2:05.94 rsync
23310 root       0 -17  578m  608  608 S  0.0  0.1   8:41.55 rsync


So I made a strace, and rsync wasn't doing much:

[root at syn1 kernel]# strace -p 23311
Process 23311 attached - interrupt to quit

(ctrl + C after one minute)

Process 23311 detached


[root at syn1 kernel]# strace -p 23310
Process 23310 attached - interrupt to quit
select(6, [5], [], NULL, {4, 800000})   = 0 (Timeout)
select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
select(6, [5], [], NULL, {60, 0}

(ctrl + C after two minutes or so)


It's in that state for several hours now.
Will it eventually wake up and finish, or perhaps something doesn't work properly?
Those "Timeout" messages in strace output worry me and make me think something didn't go as planned.

I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it looks like that to me).


-- 
Tomasz Chmielewski
http://wpkg.org