Directory cache and SMB2

[Jeremy Allison]

On Fri, Mar 30, 2018 at 01:24:10PM +0000, Tom Talpey wrote:
> > -----Original Message-----
> > From: samba-technical <samba-technical-bounces at lists.samba.org> On Behalf
> > Of Jeremy Allison via samba-technical
> > Sent: Thursday, March 29, 2018 5:23 PM
> > To: Ralph Böhme <slow at samba.org>
> > Cc: Samba Technical <samba-technical at lists.samba.org>
> > Subject: Re: Directory cache and SMB2
> > 
> > On Thu, Mar 29, 2018 at 07:47:01PM +0200, Ralph Böhme wrote:
> > > On Thu, Mar 29, 2018 at 10:12:42AM -0700, Jeremy Allison wrote:
> > > > On Thu, Mar 29, 2018 at 03:43:07PM +0200, Ralph Böhme wrote:
> > > > > Hi!
> > > > >
> > > > > It seems for SMB2 we still populate the directory cache, but we never use
> > it. Am
> > > > > I missing something?
> > > > >
> > > > > It gets populated via smbd_smb2_query_directory_send ->
> > > > > smbd_dirptr_lanman2_entry -> smbd_dirptr_get_entry -> DirCacheAdd
> > for SMB2.
> > > > >
> > > > > But the only point where it's used is via call_trans2findnext ->
> > dptr_SearchDir
> > > > > -> SearchDir which is SMB1 only.
> > > >
> > > > Yes, that's completely correct. That's because SMB1 uses a name
> > > > to resume from, wheras SMB2 always continues from the next position
> > > > (unless it's told to restart or close-and-reopen the listing).
> > > >
> > > > We could make DirCacheAdd a noop for SMB2. Wanna do a patch ?
> > >
> > > yup, I just needed the confirmation before jumping at it. :) Thanks!
> > 
> > Interesting. I just dug a little deeper, and there is indeed
> > a uint32_t SMB2_QUERY_DIRECTORY index value sent that is supposed
> > to be consulted if the SMB2_INDEX_SPECIFIED flag is sent, however
> > there's a Windows behavior note attached that says Windows servers
> > ignore this value - so this means no one will *ever* implement
> > this (might as well remove it from the spec. :-) :-).
> 
> Yeah, not all filesystems support the FileIndex argument, so historically, the protocol can't force it. The flag is, however, technically legal.
> 
> MS-FSA has this - note, "(optional)".
> 
> 2.1.5.5	Server Requests Querying a Directory
> The server provides:
> §	Open: An Open of a DirectoryStream.
> §	FileInformationClass: The type of information being queried, as specified in [MS-FSCC] section 2.4.
> §	OutputBufferSize: The maximum number of bytes to return in OutputBuffer.
> §	RestartScan: A Boolean value which, if TRUE, indicates that enumeration is restarted from the beginning of the directory. If FALSE, enumeration continues from the last position.
> §	ReturnSingleEntry: A Boolean value which, if TRUE, indicates that at most one entry MUST be returned. If FALSE, a variable count of entries could be returned, not to exceed OutputBufferSize bytes.
> §	FileIndex: An index number from which to resume the enumeration if the object store supports it (optional).

Yes, but as Windows servers don't support it, that means
Windows clients don't use it, which means "optional" is
the wrong word. "Unusable" is the right word :-).

On a meta-note, a 32-bit directory cookie is useless
these days on modern filesystems. You end up having
to keep a mapping between the real (64bit+) directory
cookie and the returned 32-bit value (I had to do
this for the old DOS SMB1 protocol level, which heavily
used this). When you have 1,000,000 file directories
that gets pretty wasteful.

The whole concept of a seekable directory cookie is
A BAD IDEA anyway (yes I know POSIX demands it) because
of race conditions - no applications can depend on
it being valid to enumerate everything in a dirctory
or even to not see entries twice or not at all. Most
filesystems internally are btrees or something more
complex.

Jeremy.