rsync freezes when copying several million files

[Paul Slootman]

On Mon 21 Aug 2006, Tomasz Chmielewski wrote:

> I just tried to copy several million files with rsync.
> 
> Unfortunately, it seems to freeze:
> 
> # rsync --delete-before -a -H -v --progress --stats /srv/backuppc-data/ /mnt/iscsi_backup/backuppc/
> building file list ...
> 7891370 files to consider
> 
> 
> And it waits here. It was already copied once sussessfully, and this time, I just wanted to do an update.
> Destination filesystem is iSCSI, on another device in LAN.
> I can list the files there, create new files etc., so it's not a device problem.
> 
> 
> As there were many files, it used up lost of RAM:
> 
> # free
>              total       used       free     shared    buffers     cached
> Mem:        524460     520928       3532          0        344       6604
> -/+ buffers/cache:     513980      10480
> Swap:      1534072    1100884     433188
> 
> But still, there is a lot of swap left.
> 
> 
> I can see that rsync is in "D" state, and doesn't use much CPU time:
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23311 root       5 -17  685m 440m  488 D  0.0 86.0   2:05.94 rsync
> 23310 root       0 -17  578m  608  608 S  0.0  0.1   8:41.55 rsync

You have 512MB of RAM, and it's using 685MB of virtual memory.


> So I made a strace, and rsync wasn't doing much:
> 
> [root at syn1 kernel]# strace -p 23311
> Process 23311 attached - interrupt to quit
> 
> (ctrl + C after one minute)

It was busy swapping in, to handle the SIGSTOP from the strace....


> Process 23311 detached
> 
> 
> [root at syn1 kernel]# strace -p 23310
> Process 23310 attached - interrupt to quit
> select(6, [5], [], NULL, {4, 800000})   = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0})       = 0 (Timeout)
> select(6, [5], [], NULL, {60, 0}
> 
> (ctrl + C after two minutes or so)
> 
> 
> It's in that state for several hours now.
> Will it eventually wake up and finish, or perhaps something doesn't work properly?

The other side is probably busy building its list of files for the
comparison.  Can you strace that process?

> Those "Timeout" messages in strace output worry me and make me think something didn't go as planned.
> 
> I tried rsync 2.6.6, and then upgraded to 2.6.8, but both froze (at least it looks like that to me).

Did you upgrade to 2.6.8 on both sides? Newer versions are better at
using less memory, although I don't know how much difference is between
2.6.6 and 2.6.8 in that respect.


Paul Slootman