[HEADS-UP] Using stored immutable file birth time for SMB File-ID

Wed Jun 26 15:46:03 UTC 2019

On Wed, Jun 26, 2019 at 6:58 AM Ralph Boehme via samba-technical
<samba-technical at lists.samba.org> wrote:
>
> Hi all,
>
> in short:
>
> Samba will adapt a new File-ID generation scheme, based on a file's initial
> creation date instead of using the file's inode number.
>
> Long story follows.
>
> Recently customers have been bitten by a problem that seriously affects
> reliabilitly of Samba as a fileserver for Mac environments.
>
> The problem is providing sane SMB File-IDs to Mac clients. We have two choices:
>
> 1) By default Samba uses file inode numbers for the SMB File-ID. As inode
> numbers may be reused, when this happens Mac client get very, very unhappy. Bug
> <https://bugzilla.samba.org/show_bug.cgi?id=12715> has some more details on the
> application level impact and more technical background.
>
> 2) To avoid the problems outlined in 1) we return 0 File-IDs to Mac clients (Mac
> clients get detected by virtue of SMB2 AAPL contect negotiation, only Macs do
> this). Returning 0 File-IDs has the same effect as setting the client side
> option "file_ids_off" on the Macs in a config file /etc/nsmb.conf to "yes". In
> both cases the client ignores the server provided File-IDs and applies some
> client side magic.
>
> Unfortunately, this mode seems to have its own serious problems:
> <https://bugzilla.samba.org/show_bug.cgi?id=13999>
>
> I've brought the issue to Apple's attention, but I don't expect a fix for this
> will be available any time soon.

I am all in favor of ignoring the bug-ridden products Apple produces ...

> This means we have to find a way to provide clients File-IDs that are never
> reused but are constantly increasing.  metze suggested somehow using a file's
> creation date, given nanosecond granularity, to minimize the risk of ID
> collisions.
>
> We can compress a struct timespec into a 64 bit File-ID with the following logic:
>
> - struct timespec.tv_nsec (nanoseconds) requires 30 bitmap, leaves 34 bit
>
> - set the high bit for creation date based File-IDs, leaves 33 bit
>
> - struct timespec.tv_sec is seconds before/after UNIX epoch, with 33 bit we can
>   encode dates up to year 2106 which should give us some time to develop
>   something even better
>
> As a file's creation time (or btime for birth-time) may be set by clients, this
> would result in changed File-IDs every time a client sets the btime. So we need
> to store an additional copy of a file's original btime. We store the btime as
> part of the DOS attributes xattr, we need an additional field in there, making
> for a new level 4 of the xattr info struct. Looking with samba-tool ntacl
> getdosinfo the new level 4 looks like this:
>
>     xattr_DOSATTRIB: struct xattr_DOSATTRIB
>         attrib_hex               : ''
>         version                  : 0x0004 (4)
>         info                     : union xattr_DosInfo(case 4)
>         info4: struct xattr_DosInfo4
>             valid_flags              : 0x00000013 (19)
>                    1: XATTR_DOSINFO_ATTRIB
>                    1: XATTR_DOSINFO_BIRTH_TIME
>                    1: XATTR_DOSINFO_CREATE_TIME
>             attrib                   : 0x00000020 (32)
>             birth_time               : Wed Jun 26 15:05:56 2019 CEST
>             create_time              : Wed Jun 26 15:05:56 2019 CEST
>
> For any file with an associated level 3 blob that lacks the birth_time field,
> when upgraded to level 4 eg when setting DOS attributes, the level 4 will have
> no birth time set:
>
>       xattr_DOSATTRIB: struct xattr_DOSATTRIB
>         attrib_hex               : ''
>         version                  : 0x0004 (4)
>         info                     : union xattr_DosInfo(case 4)
>         info4: struct xattr_DosInfo4
>             valid_flags              : 0x00000011 (17)
>                    1: XATTR_DOSINFO_ATTRIB
>                    0: XATTR_DOSINFO_BIRTH_TIME
>                    1: XATTR_DOSINFO_CREATE_TIME
>             attrib                   : 0x00000002 (2)
>             birth_time               : NTTIME(0)
>             create_time              : Wed Jun 26 15:12:40 2019 CEST
>
> Only files (and directories of course) that have an immutable birthtime will
> take advantage of the new File-IDS semantics. So generally any existing file
> will still use File-IDs based on the inode number. That shouldn't be a problem
> however, the realworld problems generally only arise with newly created files.
>
> Currently the only backend supporting storage of immutable birthtime is our DOS
> xattr via the VFS default backend. VFS modules can implement the DOS attributes
> VFS function to implement their own logic.
>
> Existing VFS modules that want to take advantage of the new File-ID semantics,
> must either implement an immutable btime natively, and until that happens they
> have to call the VFS NEXT functions to get and set the immutable btime from
> vfs_default. vfs_ceph already does this, but vfs_gpfs needs to be updated.
>
> WIP: <https://gitlab.com/samba-team/devel/samba/commits/slow-fileid> (also
> attached)
>
> Passes CI: <https://gitlab.com/samba-team/devel/samba/pipelines/68050729>
>
> Questions? Remarks? Thoughts?
>
> Thanks!
> -slow
>
> --
> Ralph Boehme, Samba Team                https://samba.org/
> Samba Developer, SerNet GmbH   https://sernet.de/en/samba/
> GPG-Fingerprint   FAE2C6088A24252051C559E4AA1E9B7126399E46

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)(传说杜康是酒的发明者)