File Fragmentation issue...especially relating to NTFS...

[Rob Bosch]

The fragmentation we see on NTFS is due to so many streams writing to the
same disk when multiple rsync clients are sending data to the rsync daemon.
Windows will not reserve space for any of the new files unless the
posix_fallocate function is used.  The writes occur as the data comes
in...resulting in extremely high fragmentation.  The testing I've done with
Matt's patch has shown the posix_fallocate function nearly eliminates the
fragmentation issue with little to no performance penalty even with very
large files (50GB+).

I'd be happy to hear the results of additional strategies though so please
post what you find out!

-----Original Message-----
From: hashproduct at gmail.com [mailto:hashproduct at gmail.com] On Behalf Of Matt
McCutchen
Sent: Sunday, December 17, 2006 8:52 AM
To: Linda Walsh
Cc: Rob Bosch; rsync
Subject: Re: File Fragmentation issue...especially relating to NTFS...

On 12/2/06, Linda Walsh <rsync at tlinx.org> wrote:
>    If rsync wants to easily encourage less fragmentation on NTFS, I'd
> suggest buffering more information in memory before doing the "write".
> Ideally, it could buffer each file in full before writing it out, but
> large files may not fit in memory.

The posix_fallocate strategy does seem to work, in Rob's setup at
least.  Perhaps you could implement the buffering strategy and run
some tests to see whether it produces fewer fragments than the
posix_fallocate strategy on various filesystems.

Matt