Fragmentation on XFS

[Jamie Lokier]

Rob Bosch wrote:
> The patch truncates the file with ftruncate if a transfer fails in
> receiver.c.  This should avoid the problem you mention.

I was thinking of a user-abort (Control-C) or crash, but this is good.

> Even if this didn't
> occur, the file would exist on the FS with the predefined size.  It would be
> in the allocation table and exist on the disk (you can see it under Windows
> explorer).  It wouldn't have data in the entire file size but it is still a
> valid, if sparse, file.  

Ok.  This is both better and worse :-)

Better than I thought, because some other filesystems (e.g. Veritas
VxFS?) allow space to be reserved without it counting in the file size,
so it's more hidden then.

Worse, because it means --prealloc will break --append --whole-file,
and the combination would sometimes be useful (e.g. for copying a
large, growing log file), and occasionally the copy is aborted by a
crash or user-abort.

> The writes in larger chunks won't fully solve the problem unless you have a
> machine that does not do much concurrency.  My Windows machine using NTFS
> experienced high fragmentation in ALL files, not just large ones.

I'm thinking large writes would fix smaller files even better than
large files, since a large write would write _all_ of a small file at
once :-)  For perspective, I'm thinking of write() calls whose size is
a megabyte or a few.  It's not the number of fragments which limits
throughput; it's the size of them.

Perhaps NTFS would not fragment individal large write() calls - so
even high concurrency wouldn't make any difference?

-- Jamie