Local disk to disk Rsync taking an hour longer than disk to remote

[Aaron W Morris]

On 7/16/07, W Smith <digitalmagnets at googlemail.com> wrote:
> Back in June I posted about the trouble I've been having backing up
> some local directories and I'm no further ahead than back then.
>
> Link for that discussion:
> http://lists.samba.org/archive/rsync/2007-June/017882.html
>
> In summary: I'm copying nearly a million small files from the main
> disk in a server to another disk in the same machine.
>
> Still on this server, but at a different time, I am backing up the
> same million files to a remote server which is completed massively
> quicker and I just can't get my head around why this should be.
>
> Here are some new stats I got whilst using "time" to measure the
> difference between the two processes.
>
> Local backup
> -------------------
> Number of files: 1060320
> Number of files transferred: 2233
> Total file size: 26814206753 bytes
> Total transferred file size: 58711290 bytes
> Literal data: 58711290 bytes
> Matched data: 0 bytes
> File list size: 47383393
> Total bytes sent: 106196771
> Total bytes received: 44680
>
> sent 106196771 bytes  received 44680 bytes  12914.54 bytes/sec
> total size is 26814206753  speedup is 252.39
>
> real    137m5.932s
> user    0m16.843s
> sys     125m35.697s
>
> Remote backup
> -----------------------
> Number of files: 1060823
> Number of files transferred: 1758
> Total file size: 28255934663 bytes
> Total transferred file size: 63027228 bytes
> Literal data: 63027228 bytes
> Matched data: 0 bytes
> File list size: 47404505
> Total bytes sent: 110514244
> Total bytes received: 35180
>
> sent 110514244 bytes  received 35180 bytes  89045.05 bytes/sec
> total size is 28255934663  speedup is 255.60
>
> real    20m40.908s
> user    0m11.344s
> sys     0m22.644s
>
>
> Sure there are a few less files on the remote backup, due to the
> scripts running at different times but this can't explain the hour
> difference in runtime can it?
>
> I've studied the rsync man page for hours, trying to find some elusive
> option to speed up local performance to no avail. I've also tried
> Unison and rdiff-backup, hoping that they might be better for local
> copying, but nothing comes close to rsync remote backup speeds.
>
> At this stage I'm close to giving up and am considering forgetting
> local backups and getting another server to have two remote backups
> instead, which wouldn't be a bad thing.

The difference could be --whole-file which is enabled by default when
the source and destination are local disks.  You could try to disable
that with --no-whole-file .

-- 
Aaron W Morris (decep)