Detecting directory changes.

Peter Smode psmode at kitsnet.vancouver.bc.ca
Fri Mar 14 20:51:47 GMT 2003


I wonder if a different approach will help until HP comes out with the OS
underpinnings we really need? From what I see, we want to use the caching
mechanism to avoid I/Os for directory information every time we do some
little operation. Caching is effective for avoiding these I/Os so long as we
actually use it and not dispose of it too often. Ideally, we would get an
AST or some discreet flag to check when further operations would take place.
Since we don't have this implemented, a compromise has been reached at a 30
second timeout.

My idea is to stick with this basic scheme, increase the rate at which we
could potentially rebuild the cache, and carry out fewer I/O. (That's the
theory at least.)

I suggest that we take a checksum of the .DIR file itself and cache that as
well. At the expiration of the timeout period, we would do another checksum
and compare. If and only if the checksums are different would we rebuild. On
any coherent VMS system, this would mean that on the vast majority of the
polls, no cache rebuild would take place. As for the code to do this, there
is the MD5 CHKSUM utility ported to VMS (VAX and Alpha) in a bunch of C
code! Sewing this in as a callable item shouldn't' be too bad (famous last
words)

This approach works since the records in the .DIR file store the filename,
version and FID. Any rename, create or delete activity will be reflected
here. Detecting changes in file allocation would continue to be detected by
the Volume Lock Value Block method.

Is this a little unsupported? Well, yes and no. We are supported if one
considers the basic property of a .DIR file and what information it must
contain to do its job. Using the checksum method, we do not care about the
layout of the .DIR file records.

This doesn't purely eliminate the problem. Also, checksum calculation is
cheap, not free. For example, check summing a 100 block .DIR file for a
directory containing 1423 files took 40 I/Os when I invoked the CHKSUM
utility from the command line as written. However, compared to the overhead
of rebuilding the cache after 30 seconds, this is a winner.

Thoughts, anyone?


Peter

P.S. I recently built v2.2.7a to look into the stat() issue. For my
installation, cache invalidation does happen with a change to space
allocation on the disk. However, the 30 second feature does not appear to be
functioning. Also, If I SMBCLIENT to myself and do a partial directory (e.g.
dir t.t) before doing a full directory, no matching files are found. Did I
go wrong, or do we have a problem?
PAS

*****************
Peter Smode
Kitsilano Network Research
psmode at kitsnet.vancouver.bc.ca


-----Original Message-----
From: samba-vms-bounces+psmode=jeslacs.bc.ca at lists.samba.org
[mailto:samba-vms-bounces+psmode=jeslacs.bc.ca at lists.samba.org]On Behalf
Of Carl Perkins
Sent: Thursday, March 13, 2003 9:47 AM
Cc: samba-vms at lists.samba.org
Subject: RE : Detecting directory changes.


>> So instead of blocking ASTs via a (probably never used) RMS resource, you
>> could do it on the F11B$s serialization resource
>
>That's exactly that resource that F11X$POSIX_FASTRDSEQNOS uses.
>The question is : is it really useful to re-write in C a routine that is
>already available in the system, even if it's an undocumented one ?

Since this could work on a VAX, but the routine doesn't exist there,
then it might be useful if you want it to work equally well on a VAX.

It also switches polling the lock value block using that routine every
time you want to read from the cache to something that is not polling
that resource. It is going from:

  Doing whatever F11X$POSIX_FASTRDSEQNOS does. This is one of
  two things - it either enqueues a lock and dequeues the lock
  in a lock mode such that it gets a copy of the current lock value
  block, or it directly accesses the memory structures used by the
  lock manager to locate and read the lock value block. Since it is
  called "FASTRDSEQNOS" and direct access would probably be faster, I
  assume that this is what it does (probably in kernel mode).

to:

  Checking a local variable. This variable is set by the blocking AST
  to indicate that the cache is invalid. This uses blocking ASTs and
  pays no attention to the sequence number in the lock value block.

I tend to prefer notification via AST over polling. It is more efficient
and should be faster: walking the lock manager's data structures (in
kernel mode, I expect) to get the data in the lock value block and
comparing it to the saved value vs. checking a local variable to see
if it has been changed from TRUE to FALSE (or whatever) by an AST.

You do still have to worry about version compatability. I would guess
that this should at least work on any version of VMS that uses the XQP,
but maybe not.

A concern is how the other, system controlled, locks behave. If they
have possible modes that indicate read or write access, then this
should work well (e.g. they could use PR mode locks when reading and
PW or EX mode locks when writing). If they don't (e.g. they just get
set to EX mode even for a read access), then it won't work so well -
it would probably require switching to using the lock value block.
If that is the case, then it is not so usefull except maybe just using
it to get the contents of the lock value block on a VAX (assuming that
it sets this, but just not provide the F11X$POSIX_FASTRDSEQNOS routine
to read it), which "just" requries taking out the lock and converting
it back and forth between modes to get the data stored in the lock value
block every time you need it. If the VAX flavor of VMS doesn't set this
value block, then it wouldn't be useful on the VAX for that and would
probably be slower than F11X$POSIX_FASTRDSEQNOS on the Alpha (and thus
not useful there either).

So the usefullness depends on the behavior of VMS's use of these locks
and also whether or not the VAX version uses the lock value block to
store sequence number information.

--- Carl
PLEASE READ THIS IMPORTANT ETIQUETTE MESSAGE BEFORE POSTING:

http://www.catb.org/~esr/faqs/smart-questions.html



More information about the samba-vms mailing list