Local disk to disk Rsync taking an hour longer than disk to
digitalmagnets at googlemail.com
Thu Jul 19 14:13:16 GMT 2007
Sincere thanks to everyone for their replies. Before following some of
the advice offered I decided to investigate the machine filesystem
I'm no linux expert (obviously :)) but found that of the ext3
filesystems on each disk, the dir_index feature was only enabled on
the primary disk. Research revealed that this helped with directory
listing and it made sense that with 1mill files in one directory, it
might be a good idea to have it enabled on both disks!
Result: The local backup completes in <10min instead of 1hr20min+.
It also explains why local primary disk to remote primary disk backup
was fast, as both these had dir_index enabled. On CentOS at least,
additional mounted ext3 drives don't seem to have dir_index enabled by
It's a massive relief to have this finally sorted, I've spent far too
many hours trying to fix it. I don't know whether this might be a
useful tip in the Rsync FAQ troubleshooting when backing up large
directories of small files.
More information about the rsync