Slow backup

[Fabian Cenedese]

>>>  I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
>>> 
>>>In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. 
>>>When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours. 
>>>It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
>>>Does anybody encounted cases like this? Any help will be thankful.
>>
>>If this is just one directory on one hard disk then the disk head has to
>>move constantly from one place to another to another to another for
>>each process. It's possible that it will run even faster if you have
>>just one process. On the other hand Windows will cache file info
>>in RAM so a second run could be faster then the first one. This
>>depends one your RAM, how much have you got? Have you tried
>>with just one process?
>
>Thanks for your reply. I have tried one process, but have no improvement in essence. So the question is
>why the file comparison consumes so much time? In my mind, building file list may consumes some time,
>but comparison should very fast, what is the problem?

Please keep it on the mailing list. I'm not an expert, others might
know more than me.

rsync has to stat each file and directory. I don't know what exactly
takes much time. But maybe you can give more information about
your setup: What OS, what file system, how many files and dirs,
small/big files, connection speed, full command line with all
rsync options, RAM, CPU etc.

Are you using cwRsync? With the CygWin emulation? This
is known to be slower. But even then 5h seems slow.

Just as an example: Here's a log from a backup running on a
slow NAS to another NAS in the local LAN not doing much
as the files are already the same. But both are running Linux
or actually BusyBox:

Number of files: 3445179
Number of files transferred: 0
Total file size: 349083766943 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 71253503
File list generation time: 0.016 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71953972
Total bytes received: 700464

sent 71953972 bytes  received 700464 bytes  66079.52 bytes/sec
total size is 349083766943  speedup is 4804.71

real    18m18.760s
user    0m55.883s
sys     2m10.016s

So over 3 million files in 20 minutes. What are your --stats?

bye  Fabi