Fragmentation on XFS
jamie at shareable.org
Mon Feb 25 22:35:24 GMT 2008
Rob Bosch wrote:
> The patch truncates the file with ftruncate if a transfer fails in
> receiver.c. This should avoid the problem you mention.
I was thinking of a user-abort (Control-C) or crash, but this is good.
> Even if this didn't
> occur, the file would exist on the FS with the predefined size. It would be
> in the allocation table and exist on the disk (you can see it under Windows
> explorer). It wouldn't have data in the entire file size but it is still a
> valid, if sparse, file.
Ok. This is both better and worse :-)
Better than I thought, because some other filesystems (e.g. Veritas
VxFS?) allow space to be reserved without it counting in the file size,
so it's more hidden then.
Worse, because it means --prealloc will break --append --whole-file,
and the combination would sometimes be useful (e.g. for copying a
large, growing log file), and occasionally the copy is aborted by a
crash or user-abort.
> The writes in larger chunks won't fully solve the problem unless you have a
> machine that does not do much concurrency. My Windows machine using NTFS
> experienced high fragmentation in ALL files, not just large ones.
I'm thinking large writes would fix smaller files even better than
large files, since a large write would write _all_ of a small file at
once :-) For perspective, I'm thinking of write() calls whose size is
a megabyte or a few. It's not the number of fragments which limits
throughput; it's the size of them.
Perhaps NTFS would not fragment individal large write() calls - so
even high concurrency wouldn't make any difference?
More information about the rsync