File Fragmentation issue...especially relating to NTFS...

[Linda Walsh]

Hey Rob, some clarifications/comments...:-)

I was talking about the undesirable fragmentation that results from using
rsync as a type of smart-copy -- copying files from a source to a
destination with the target "rsync" invoked by the source-tree process (i.e.
rsync isn't a daemon). For me, that's the most common case.  Rsync and other
cygwin-based utils don't operate as well as the native Windows Copy or Xcopy
commands.

I never use rsync in daemon mode due to security paranoia.  I use 'ssh' when
connecting to remote systems.  I do use rync with "RSYNC_RSH =/usr/bin/ssh".
It is also rare that I have multiple rsyncs copying to the same target
system or have multiple rsync commands copying/updating directory trees on
the same system, locally.

My test case, that produced bad fragmentation, was using only 1 writer in
the case for rsync copying from one source host to another.  It came out of
this same topic (File Fragmentation) arising on the cygwin list, where there
was a general problem with the *nix utils producing poor fragmentation
performance on NTFS.

I did tests of two linux file systems that had utils that listed #fragments
on their file system (only ext2/3 and xfs had such a util).  I used
sysinternals "Contig" program to show fragmentation on NTFS.  Writing the
entire in 1 large write (using a 64M test file), provided the fewest
fragments on NTFS.  The write size didn't make as much of a difference on a
heavily fragmented ext2/ext3 disk, with over 500 fragments being produced
for the 64M file in my test cases.  The write size also didn't make as much
of a difference on XFS: it reliably put the 64M file in 1 contiguous
fragment (extent) in all of my tested block sizes.

These were all cases using 1 writer/disk.

For NTFS, what seemed to be true was that fragmentation could be positively
affected by writing output files in 1 write call -- buffering the entire
file in memory before writing.

For example, if I use rsync to copy 1 file to a target host where the target
host uses NTFS, rsync writes in "smallish" chunks (better than the gnu-file
utils, though).  As a result, NTFS looks for a contiguous space for each of
the "smallish" chunks (whatever rsync's file-write size is).

In this simplistic case, on NTFS, rsync can achieve performance as optimally
as the native windows utilities like the shell Copy and xcopy commands
internet explorer by buffering the entire file in memory and doing the final
write in 1 write call (this presumes your file fits in memory, which is
another "issue").

In the single-writer case, there doesn't seem to be any performance penalty
that I could easily measure.  As an aside, IE downloads a file first to a
tempdir.  Given slow download speeds compared to other operations, it might
have been presumed that users might engage in other activities while a large
file download was in progress -- this would approximate the multiple-writers
case.

As for multiple writers -- one might have to use the falloc type call, which
may be more effective for a heavily loaded "server" process writing multiple
files at the same time, as it might be too memory intensive to buffer all
incoming files completely in memory before writing them out.

I do wonder, though, since the "falloc" type call (or windows equivalent) is
presumably more efficient than downloading to a tmpdir and then copying to
the final destination, why wouldn't the MS ID designers have used the call?
MS has a history of using internal or specialized MS calls to make their
products perform better than competitors.  I'd think the "falloc" call would
be a perfect candidate for speeding up file download performance in their
browser (in the sense that it is a type of file-transfer "agent").  Maybe
they were just being lazy?


Rob Bosch wrote:
 > The fragmentation we see on NTFS is due to so many streams writing to the
 > same disk when multiple rsync clients are sending data to the rsync daemon.
 > Windows will not reserve space for any of the new files unless the
 > posix_fallocate function is used.  The writes occur as the data comes
 > in...resulting in extremely high fragmentation.  The testing I've done with
 > Matt's patch has shown the posix_fallocate function nearly eliminates the
 > fragmentation issue with little to no performance penalty even with very
 > large files (50GB+).
 >
 > I'd be happy to hear the results of additional strategies though so please
 > post what you find out!
---

	I would be too! :-)

	Seriously -- I think I need to get some more HW here, my previously used
"test disk" has gotten a bit too full to be real useful.

	I'm not sure I'm quite as interested in the multi-writer case from a
practical standpoint: I don't see too many cases where a single user using
rsync is going to see this behavior.  Additionally, even on a server, which
might be the target of multiple writers, do you have any information on how
often one actually sees multiple writers?  Statistically, I wouldn't think
it would be often.

	_If_ you agree that it isn't the common or likely case, I'd also be
concerned about how much (if it does "at all") it affects throughput.  Rsync
is, unfortunately slower than either cp or tar when the target tree is not
present.  I don't have figures at hand, but I seem to remember it being
20-30% slower than either in local file-transfers, and notably slower than
"tar <input> |ssh remote-sys tar <output>" (with rsync using RSH=ssh).

	While I would guess that rsync is used more often to update (vs.
create), I think it would be "great", if it had performance in "create"
cases that had "worst" cases only 5% worse than cp or tar (though preferably
0% worse, or _better_, (assuming it's possible; may not be).

	Rsync's speed in "create" cases vs. cp and tar is more often an issue
for me than file-fragmentation, as I have defragmentation run nightly.

	I.e. -- "bottom" line -- does "fadvise" have a negative performance
impact?  In my usage, that would be less desirable than concerns about
fragmentation (which get addressed by nightly defrags).

Linda