rsync freezes when copying several million files

[Tomasz Chmielewski]

> On Mon 21 Aug 2006, Tomasz Chmielewski wrote:

(...)

>> Process 23311 detached
>>
>>
>> [root at syn1 kernel]# strace -p 23310
>> Process 23310 attached - interrupt to quit
>> select(6, [5], [], NULL, {4, 800000})   = 0 (Timeout)
>> select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
>> select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
>> select(6, [5], [], NULL, {60, 0}
>>
>> (ctrl + C after two minutes or so)
>>
>>
>> It's in that state for several hours now.
>> Will it eventually wake up and finish, or perhaps something doesn't work properly?
>
> The other side is probably busy building its list of files for the
> comparison.  Can you strace that process?

There is no "other side":

rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ /mnt/iscsi_backup/backuppc/

It's not over SSH or anything similar.

/mnt/iscsi_backup/ is /dev/sda device, mounted only on that machine, used by rsync only:

# lsof -n|grep /mnt/iscsi_backup
rsync     23310      root  cwd       DIR        8,0      4096          2 /mnt/iscsi_backup
rsync     23311      root  cwd       DIR        8,0      4096   21774337 /mnt/iscsi_backup/backuppc


>> Those "Timeout" messages in strace output worry me and make me think something didn't go as planned.
>>
>> I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it looks like that to me).
>
> Did you upgrade to 2.6.8 on both sides? Newer versions are better at
> using less memory, although I don't know how much difference is between
> 2.6.6 and 2.6.8 in that respect.

Both versions built the file list consisting of almost 8 million files relatively fast (under an hour I think), after that, I'm waiting for the 4th
hour right now, and nothing seems to happen.
The system is swapping heavily, as I used -H options, and there are lots of hardlinks there:

# iostat /dev/md0  # swap partition
Linux 2.6.16-xen (syn1)         21.08.2006

avg-cpu:  %user   %nice %system %iowait   %idle
          18,84    3,40   11,85   17,96   47,96

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
md0              81,14       497,92       151,23  127504024   38725232


Perhaps I should check back in the morning, maybe it'll finish by then?

If yes, I'll give some numbers on how long did that take; if it will not finish, I'll complain some more :)


-- 
Tomasz Chmielewski
http://wpkg.org