Local disk to disk Rsync taking an hour longer than disk to remote
digitalmagnets at googlemail.com
Mon Jul 16 20:35:46 GMT 2007
Back in June I posted about the trouble I've been having backing up
some local directories and I'm no further ahead than back then.
Link for that discussion:
In summary: I'm copying nearly a million small files from the main
disk in a server to another disk in the same machine.
Still on this server, but at a different time, I am backing up the
same million files to a remote server which is completed massively
quicker and I just can't get my head around why this should be.
Here are some new stats I got whilst using "time" to measure the
difference between the two processes.
Number of files: 1060320
Number of files transferred: 2233
Total file size: 26814206753 bytes
Total transferred file size: 58711290 bytes
Literal data: 58711290 bytes
Matched data: 0 bytes
File list size: 47383393
Total bytes sent: 106196771
Total bytes received: 44680
sent 106196771 bytes received 44680 bytes 12914.54 bytes/sec
total size is 26814206753 speedup is 252.39
Number of files: 1060823
Number of files transferred: 1758
Total file size: 28255934663 bytes
Total transferred file size: 63027228 bytes
Literal data: 63027228 bytes
Matched data: 0 bytes
File list size: 47404505
Total bytes sent: 110514244
Total bytes received: 35180
sent 110514244 bytes received 35180 bytes 89045.05 bytes/sec
total size is 28255934663 speedup is 255.60
Sure there are a few less files on the remote backup, due to the
scripts running at different times but this can't explain the hour
difference in runtime can it?
I've studied the rsync man page for hours, trying to find some elusive
option to speed up local performance to no avail. I've also tried
Unison and rdiff-backup, hoping that they might be better for local
copying, but nothing comes close to rsync remote backup speeds.
At this stage I'm close to giving up and am considering forgetting
local backups and getting another server to have two remote backups
instead, which wouldn't be a bad thing.
My last hope is in the wisdom of this mailing list... :)
More information about the rsync