Samba + exFAT : how to avoid pre-allocating when copying big files?
j at gget.it
Wed Dec 9 13:50:49 UTC 2020
> FileEndOfFileInformation == ftruncate();
> FileAllocationInformation == fallocate(.., FALLOC_FL_KEEP_SIZE, offset,
Jeremy, by ftruncate() do you mean "immediate writing of null bytes"?
I've analyzed syscall logs to see how some programs on Windows do, which
1) create a file on exFAT partition
2) immediately set 1 GB file size *without* any delay; it's instantaneously
reported as 1GB in the Explorer; also, there is no 1GB null-bytes writing
3) then write the actual file content
It goes like this:
t=0.000s: SetFileInformationByHandle with FILE_END_OF_FILE_INFO,
with FILE_ALLOCATION_INFO, AllocationSize=1073741824
t=0.001s WriteFile (here the real data writing begins)
t~44sec WriteFile (last block written)
So this confirms that, with the Windows native exFAT driver, neither
"FileEndOfFileInformation" nor "FileAllocationInformation" triggers an
immediate truncate() or writing of null bytes for an opened file handle.
This "writing of null bytes" only happens when the file is closed for the
"remaining unwritten blocks", that's the trick. Example:
CreateFile, SetEndOfFileInfo+SetAllocationInfo 1GB, Close => then 1GB
of null bytes written
CreateFile, SetEndOfFileInfo+SetAllocationInfo 1GB, Write 100MB, Close
=> then only 900 MB of null bytes written
CreateFile, SetEndOfFileInfo+SetAllocationInfo 1GB, Write 500MB, Close
=> then only 500 MB of null bytes written
CreateFile, SetEndOfFileInfo+SetAllocationInfo 1GB, Write 1000MB,
Close => then 0 MB of null bytes written ! <-----
Namjae, do you think this should be done directly at the Linux exFAT driver
I'm not sure if the modification should be at the Linux exFAT driver layer,
or in Samba?
If the latter, in Samba, would it be possible to avoid that
SetFileEndOfFileInformation == ftruncate() == immediate writing of null
TL;DR: with Windows' native exFAT driver, SetAllocation+SetEndOfFileInfo
does not write any null bytes now, if the file handle is still open. It
will only write null bytes, on file close, on the *remaining* space that
has not been written before. If all the preallocated blocks are already
written, there is no null-bytes writing.
More information about the samba-technical