inotify on Samba4 and clustering

[James Peach]

On Tue, 2006-05-30 at 16:38 -0500, Abhijith Das wrote:
> Hi list,
> I was looking at the latest Samba4 code and found some use of the 
> inotify interface. I was told before that dnotify was being replaced by 
> inotify, so this is not a surprise.
> However, when I think of a clusterized implementation of inotify for 
> GFS, I see complications.
>  - An event-watch on one cluster node should be propagated to all nodes.
>  - When an event occurs on a different node, we need to somehow write to 
> the event-queue of the first node. This could mean some serious hacking.
>  - There are way too many events to use a distributed-lock-manager's 
> locking for internode communication. A separate userland event/messaging 
> framework would have to deal with these event notifications and pass 
> them around as necessary.

If you have a cluster architecture where a single node arbitrates
metadata access, you can have a system where the notification is applied
to the arbiter and the arbiter triggers the notification on behalf of
clients. I think there will might be some holes in this scheme since
clients can do I/O without notifying the arbiter immediately. It should
be a 90% solution, though.

> Is the effort worth the reward? In what situations does samba use 
> inotify and what could be the performance impact without inotify support?

inotify is there to support the FindFirstFileChangeNotification call. As
Andrew says, this is a correctness issue, and there may be apps that
depend on this call working correctly. In a cluster it might be OK to
skip it if there is a known set of qualified apps running.

> An alternate solution could be a local-inotify implementation. We could 
> add support in clustered-samba to pass these inotify messages to various 
> cluster nodes as appropriate?

This might be a reasonable option. You already need something similar
for clusters that don't support oplocks. You'd need to register a watch
with the locking leader who would then force all the nodes to apply a
local watch .. or something like that.

-- 
James Peach | jpeach at samba.org