Local disk to disk Rsync taking an hour longer than disk to remote

[W Smith]

Back in June I posted about the trouble I've been having backing up
some local directories and I'm no further ahead than back then.

Link for that discussion:
http://lists.samba.org/archive/rsync/2007-June/017882.html

In summary: I'm copying nearly a million small files from the main
disk in a server to another disk in the same machine.

Still on this server, but at a different time, I am backing up the
same million files to a remote server which is completed massively
quicker and I just can't get my head around why this should be.

Here are some new stats I got whilst using "time" to measure the
difference between the two processes.

Local backup
-------------------
Number of files: 1060320
Number of files transferred: 2233
Total file size: 26814206753 bytes
Total transferred file size: 58711290 bytes
Literal data: 58711290 bytes
Matched data: 0 bytes
File list size: 47383393
Total bytes sent: 106196771
Total bytes received: 44680

sent 106196771 bytes  received 44680 bytes  12914.54 bytes/sec
total size is 26814206753  speedup is 252.39

real	137m5.932s
user	0m16.843s
sys	125m35.697s

Remote backup
-----------------------
Number of files: 1060823
Number of files transferred: 1758
Total file size: 28255934663 bytes
Total transferred file size: 63027228 bytes
Literal data: 63027228 bytes
Matched data: 0 bytes
File list size: 47404505
Total bytes sent: 110514244
Total bytes received: 35180

sent 110514244 bytes  received 35180 bytes  89045.05 bytes/sec
total size is 28255934663  speedup is 255.60

real	20m40.908s
user	0m11.344s
sys	0m22.644s


Sure there are a few less files on the remote backup, due to the
scripts running at different times but this can't explain the hour
difference in runtime can it?

I've studied the rsync man page for hours, trying to find some elusive
option to speed up local performance to no avail. I've also tried
Unison and rdiff-backup, hoping that they might be better for local
copying, but nothing comes close to rsync remote backup speeds.

At this stage I'm close to giving up and am considering forgetting
local backups and getting another server to have two remote backups
instead, which wouldn't be a bad thing.

My last hope is in the wisdom of this mailing list... :)