Samba + exFAT : how to avoid pre-allocating when copying big files?

ronnie sahlberg ronniesahlberg at gmail.com
Tue Dec 8 03:40:11 UTC 2020


On Tue, Dec 8, 2020 at 1:20 PM Namjae Jeon via samba-technical
<samba-technical at lists.samba.org> wrote:
>
>
>
> > -----Original Message-----
> > From: Richard Sharpe <realrichardsharpe at gmail.com>
> > Sent: Tuesday, December 8, 2020 11:21 AM
> > To: Namjae Jeon <namjae.jeon at samsung.com>
> > Cc: Joseph <j at gget.it>; samba-technical <samba-technical at lists.samba.org>
> > Subject: Re: Samba + exFAT : how to avoid pre-allocating when copying big files?
> >
> > On Mon, Dec 7, 2020 at 6:12 PM Namjae Jeon via samba-technical <samba-technical at lists.samba.org> wrote:
> > >
> > > > Hello,
> > > > When sending a big file (for example 2 GB) from a Windows computer
> > > > to a Linux computer (with Samba server), then:
> > > > * if the destination FS is ext4, NTFS, no problem
> > > > * if the destination FS is exfat, then there is a strange delay (~30
> > > > seconds) during which the Windows Explorer waits, before actually
> > > > copying the file It seems that the Samba server first "preallocates" the 2GB, before accepting to
> > start the copy from Samba client.
> > > > Thus, 2GB (of probably null bytes?) are first written, and then, the actual file is written.
> > > > How to avoid this effect that doubles the number of gigabytes written and doubles the transfer
> > time?
> > > >
> > > > I have already tried:
> > > >
> > > >     strict allocate = no
> > > >
> > > > without any success in the exfat case.
> > > >
> > > > Any idea?
> > > Windows seems to send SMB_FILE_END_OF_FILE_INFORMATION smb2 setinfo
> > > request to pre-allocate space as much as the file size before writing the file.
> >
> > I believe that Windows does this because applications are actually writing into the Windows equivalent
> > of the page cache.
> >
> > To avoid problems where the server runs out of space when the cache flushes, they send a request to
> > extend the file to the new EOF when a write to a larger offset occurs. That way, the ENOSPC error can
> > be returned on the WRITE rather than on the CLOSE because no one checks the error return from a close
> > and anyway, how can you tell which write failed when you get an error on close?
> I didn't want to remove ftruncate before writing in samba. I mean that there is
> an improvement if exfat fallocate is implemented. One misunderstanding is that
> the FileEndOfFileInformation request must change the size of the file.
> Because of that, We may not use keep size flags of fallocate(). BTW, Why do
> windows call FileEndOfFileInformation instead of FileAllocationInformation
> of smb2 setinfo before writing?

They have slightly different semantics. FileEndOfFile both sets end of
file and also allocates all the blocks
(for the 99% of cases where the files are non-sparse) while
FileAllocationInformation only allocates the blocks
but does not set the end-of-file, and also once the file is closed,
any blocks past end-of-file allocated with FileAllocationInformation
would be deallocated and returned.

They have always used FileEndOfFile, maybe they are just afraid of
changing the semantics and that this might "surprise"
applications.


>
> Thanks!
> >
> > > samba fallback to call ftruncate if fallocate is not support in local filesystem.
> > > I personally think that it is possible to improve the problem if exFAT
> > > support keep size flags of fallocate(). It doesn't support yet in
> > > linux kernel exFAT, but I have a plan. exFAT can store the number of
> > > allocated cluster size and the current file size in metadata of file.
> > > Unlike FAT. so it seems that the fallocate can be implemented properly in exFAT.
> > > >
> > > > PS:
> > > > * I've already tried exfat-fuse, but also exfat-nofuse, and also
> > > > exfat (non
> > > > fuse) included in Kernel 5.7, and it's the same.
> > > > * There are many forums posts with the same problem, e.g.
> > > > https://www.raspberrypi.org/forums/viewtopic.php?t=206866
> >
> > --
> > Regards,
> > Richard Sharpe
> > (何以解憂?唯有杜康。--曹操)(传说杜康是酒的发明者)
>
>
>



More information about the samba-technical mailing list