[Samba] IIS Change Notifications

Sun Jan 9 00:31:11 MST 2011

Since I posted the above, i've done more testing and research.

Using my binary test files, I've captured a log from smbd as I change files
on disk - these changes don't reach IIS.
I'm pretty sure this is the relevant portion where the notification fires:

inotify_dispatch called with mask=2, name=[ChangeNotifyTesting.dll]
inotify_dispatch: ne.action = 3, ne.path = ChangeNotifyTesting.dll
filter_match: e->mask=2, w->mask=210003c6, w->filter=17
sys_notify_callback called with action=3, for ChangeNotifyTesting.dll
notify_callback called for client/site.co.nz/htdocs/Test/bin
    0000 offset: 00000000
    0004 action: 00000003
    0008 namelen: 0000002e
    000c name: C.h.a.n.g.e.N.o.t.i.f.y.T.e.s.t.i.n.g...d.l.l...
Client only wanted 32 bytes, trying to marshall 58 bytes
size=71

--snip-- (looks like the notification is set up again after it's fired)

[0000] 17 00 00 00 D8 2A 00 00                            .....*..
call_nt_transact_notify_change
call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|ATTRIBUTES|LAST_WRITE,
recursive = 0
Locking key 6E6F7469667920617272
Allocated locked data 0x0x7f9cb4c5e450
inotify_add_watch for
/srv/web_store/microsoft/client/site.co.nz/htdocs/Testmask 210003c6
returned wd 19
Unlocking key 6E6F7469667920617272
change_notify_add_request: Adding request for client/site.co.nz/htdocs/Test:
max_param = 32

Taking this a bit further, i've eliminated IIS entirely for the time being,
with a C# application using FileSystemWatcher (available if a samba
developer would like a copy to test with)
watching the UNC path;
I find the following:

Modifications of files show information in the samba log, but do not reach
the FileSystemMonitor
Deleting a file successfully reaches FileSystemMonitor
Creating a file successfully reaches FileSystemMonitor
Moving a file into another (overwrite) does not reach FileSystemMonitor
(same as modification)

Here's the output;
commands over SSH are prefixed with >,
comments with //,
log output with *,
Monitor App with $

Registered ChangeNotifications for
\\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test;
Recursive=True
> touch fish
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
* Transaction 24 of length 88 (0 toread)
* switch message SMBnttrans (pid 1366) conn 0x7f0f88645290
* setting sec ctx (1001, 1001) - sec_ctx_stack_ndx = 0
* call_nt_transact_notify_change
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish Created
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish Changed
> nano fish
//edit file and press ctrl-o, ctrl-x
$ No Output
* No Output
> cp ../Green/bin/ChangeNotifyTesting.dll fish
$ No Output
* No Output
> rm fish
* Transaction 25 of length 88 (0 toread)
* switch message SMBnttrans (pid 1366) conn 0x7f0f88645290
* setting sec ctx (1001, 1001) - sec_ctx_stack_ndx = 0
* call_nt_transact_notify_change
* call_nt_transact_notify_change: notify change called on client/
site.co.nz/htdocs/Test, filter = FILE_NAME|DIR_NAME|LAST_WRITE|LAST_ACCESS,
recursive = 1
$ File: \\smbtest\mwh-webstore\client\site.co.nz\htdocs\Test\fish Deleted

Recursive true/false makes no difference (I was making the changes in the
folder I had the notification registered for)
(-r argument on testing app)

FileSystemWatcher internally uses the ReadDirectoryChangesW API function:
http://msdn.microsoft.com/en-us/library/aa365465%28v=vs.85%29.aspx
And some googling reveals that ReadDirectoryChangesW doesn't work with
samba, and quite a few people describe this issue.

Some more googling yeilded that IIS makes heavy use of ReadDirectoryChangesW
internally, so this is quite likely the root cause of the issue.

This blog post suggests that there are some errors returned, and also
describes something I saw in testing with IIS which is after a while, the
notifications just stopped firing completely- I think this is due to
repeated errors being returned, and as a result, the notification system is
shut down:
http://leontheprogrammer.blogspot.com/2010/04/readdirectorychangesw-returns.html

This is a post from 2003, where it was reported that ReadDirectoryChangesW
wasn't working and a workaround, but unfortunately, because I can't modify
anything in IIS itself (or FileSystemWatcher for that matter) it's not the
correct solution here.
http://lists.samba.org/archive/samba-technical/2003-July/030902.html

I don't think there's much more testing I can do here- it seems like the
_changes_ to the folder never show up in samba, and as a result, don't get
pushed to the watcher; i'm only using debug 3; 10 is a bit hard to read :)

I used inotify-tools and manually verified that the expected inotify events
fire when I perform the various operations.
That is, I performed the above commands and found that the inotifywait (I
also used inotifywatch but I couldn't get line by line output as changes
happened with this one) and observed the expected sequence of CREATE, OPEN,
MODIFY, DELETE was raised.
This makes me think that when Samba receives the change notification from
ReadDirectoryChangesW, it doesn't successfully either
a) decode the request parameters
b) translate the request parameters to the underlying inotify arguments (and
thus those changes don't get monitored)

For completeness' sake, I forced recursive to true in nttrans.c and then
tested - I can confirm that the behaviour/output did not change.

I hope this is enough information for someone to either investigate the
issue or to ask for more information regarding a specific test

Gareth
Sniper Systems Ltd
New Zealand

On Sun, Jan 9, 2011 at 3:13 PM, Gareth Evans <agrath at gmail.com> wrote:

> Good afternoon all,
>
> I'm a web developer (c#) who also manages a small hosting environment.
> Roughly, our configuration is:
>
> Two IIS7 (Windows Server 2008 Web) servers which share configuration data
> and site files from a SMB share located on a ubuntu server.
>
> The ubuntu server is currently running 3.0.28a which is quite old, but read
> on.
>
> We've been having a problem with the change notifications from the ubuntu
> filesystem reaching IIS.
>
> The symptoms are that if a file is changed on disk (such as a binary/dll or
> web.config or similar) are changed, IIS does not flush the appdomain and
> reload these files, it continues on using the old files until either
>
> 1) an app pool reset is performed (typically bad for performance and not
> something you should do all the time)
> 2) samba is restarted (definitely not recommended)
> 3) the file that was changed is deleted and then added again
>
> My test case is a simple .net based application which uses code to set the
> colour of a box.
> I compiled three different binaries, one for red/green/blue.
>
> I can then switch the binaries out with a simple cp
> (e.g. cp red/bin/* test/bin - where test is the wwwroot for my test site)
>
> When I update the binary, IIS doesn't reload it.
>
> I have tested many different configurations.
>
> IIS7 accessing XP file share - works
> IIS7 accessing Win7 file share - works
> IIS7 accessing 2003 file share - works
> IIS7 accessing 2008 file share - works
> IIS7 accessing ubuntu samba 3.0.28a share - doesn't work
> IIS7 accessing ubuntu samba 4.?? beta share - doesn't work
> IIS7 accessing ubuntu samba 4 running SMB2 experimental - doesn't work
> IIS7 accessing ubuntu 3.5.6 (built from source) share - doesn't work
>
> Basically, if the share provider is windows (SMB1 or SMB2), it works - if
> it's linux/samba it doesn't.
>
> I've done a lot of reading on this topic, and I'm pretty sure that IIS uses
> a different change notification mechanism to that of your standard file
> browser window; that is, if I have a explorer window looking at the share, I
> see the modify date change immediately, however IIS doesn't realise and load
> the new binary.
>
> I've done some debugging and at one point I saw messages from inside
> notify.c; Client only wanted %d bytes, trying to marshall %d bytes\n; which
> results in notify_marshall_changes returning false.
>
> As far as I can tell, this bug has existed for quite some time and probably
> hasn't been diagnosed correctly.
>
> I tested explicitly today using a windows 7 desktop (IIS7 installed from
> programs and features) with both a local folder (to first confirm that my
> binaries would work as expected then a freshly installed 10.10 ubuntu with
> samba 3.5.4 (from launchpad/aptitude) followed by a built from source 3.5.6.
>
> I tested with 3.5.6 to confirm it wasn't the 4 byte boundry bug recently
> fixed.
>
> The ubuntu server is running on a VM locally.
>
> IIS7 works much smoother with UNC paths than IIS6, except you have to use
> caspol to trust the UNC path if you want to execute asp.net binaries.
>
> I'm not very familiar with C code, especially on linux environments so i'm
> not sure how to test/provide the information you need to diagnose this bug.
>
> I am happy to file a bug report, but without a working IIS7 installation,
> it will be very hard to replicate.
> It's not reasonable to expect the samba developers to install and test an
> IIS7 install so I would be happy to work with someone on the problem.
>
> I can verify the problem in the most simple test case, and I have tested a
> reasonable subset of samba installs.
>
> My thoughts are it's something to do with watching a folder of files, and
> this not propogating correctly - as in my testing the notifications reached
> samba and appeared to be sent to windows, just IIS didn't respond to them.
> This thread from 2003 describes a very similar issue with classic ASP (the
> precursor to .net)
> http://www.mail-archive.com/samba@lists.samba.org/msg12479.html
>
> This issue would be great to fix, as it's pretty much the only stumbling
> block in using SMB as a file share for IIS, we've been doing this for about
> a year now - and it works perfectly once you set it up - with the exception
> of the change notifications.
>
> We've been working around it by recycling application pools but as
> mentioned above, this isn't really a long-term solution and it means 3rd
> party developers need way more access to the hosting environment (to recycle
> pools etc) than I consider practical.
>
> Appreciate any advice/help - and happy to provide more debugging
> information when required,
>
> Gareth Evans
> Sniper Systems Ltd
> New Zealand
>