[Samba] Custom VFS

Andrew Scherpbier andrew at scherpbier.org
Wed Jul 25 09:54:40 MDT 2012

On 07/24/2012 11:37 PM, ivenhov wrote:
> Andrew.
> On 24 July 2012 22:35, Andrew Scherpbier <andrew at scherpbier.org> wrote:
>> The strategy I've used is to write a simple TCP protocol client (the VFS
>> module) and server (a straight forward threaded Java server).
>> Works like a charm.
> I was considering pretty much the same technique using localhost and
> sockets to separate native library from Java service.
>> As long as the client side is abstracted enough so that its samba
>> connection state is independent from the server connection state, there are
>> no issues with restarting either.  (I started out using a statefull
>> protocol, but ended up changing to a completely stateless one, where the
>> individual messages contain enough information to establish context.  This
>> way, if either end of the system goes down, recovery is the simple act of
>> building a new TCP connection.)
> That means you need to carry enough information to resume and also have
> some sort of queue of messages on the client (VFS) that has not been
> delivered yet.
> Is that correct?

Well, this was actually my main reasoning behind using ActiveMQ as the 
comm layer initially.  However, it turned out that since what I needed 
to pass onto the Java service actually required acknowledgements, the 
whole queuing up of requests wasn't actually used!  (20-20 Hindsight!)
So it was pretty simple to put enough information in each request to 
make the whole thing stateless.
Mind you, there is state kept in the VFS module to track open files, but 
for the current project the java service has no need to know about file 
descriptors; it only cares about whole files.

> I originally looked into hosting the JVM in the VFS module, but that was
>> going to be a problem because each smbd process would have to start its own
>> JVM.  The JVM startup time (especially the server JVM) is very high and the
>> memory overhead would not make it scalable.
> Why do you need several smbd on single host? Is it because of high
> availability or some latency issues you wanted to remove?
> Or did I misinterpreted that?

Because each connection to samba creates a new smbd process.  So 100 
clients == 100 smbd processes.   (here I'm saying a client is a windows 
computer...  multiple programs running on a single windows computer 
under the same profile will share the connection.)

Nothing wrong with that.  That's just the way Samba works.

>> The system I'm working on now manages PB class storage (currently up to
>> 10PB) with hundreds of concurrent clients and the VFS module does this
>> without issues or much overhead.  We're regularly seeing write speeds in
>> the 400-500MB/s range using 10GbE and multiple windows clients.
> Do you use hot-standby Samba server for failover, clustered Samba etc? If
> yes, how do you achieve that if you don't mind telling?

Our system uses active-passive failover using heartbeat.  We're not 
clustering, although that's on the roadmap.  We're using heartbeat 
simply because it is already well supported by the OS and its management 
that we're using.
So we use some straight forward FC storage for database and local 
storage and heartbeat takes care of the switching.
For the current target market (Video Surveillance) this setup works well 
enough since there are going to be only a limited number of clients that 
write continuously and that can deal with the small hiccup that occurs 
when a failover happens.

> And quick question about notification. If in your system file appears or
> was modified outside Samba, is there a way of notifying Samba clients about
> that change?
That's an excellent question and one that we've discussed a lot 
internally.  The design decision ended up stating that we will only 
support client access through Samba.
As a simple safeguard, however, when the Java service starts, it kicks 
off a filesystem scanner running under ionice idle class to see if 
somehow there were any files created outside of its knowledge.

If our clients end up wanting NFS access to our files, we'll have to 
switch from using VFS to using something like FUSE or create our own 
kernel module.  Our current solution for those clients is to tell them 
to use smbmount.  :-)

> Notification that goes from VFS layer so Samba and then to Windows clients
> to refresh directory, Explorer view etc.
> I think that mechanism exists in Samba via inotify but I may be wrong, I'm
> Samba newbie.

The default_vfs takes care of all that using oplocks, etc.  (Please 
correct me if I'm wrong, samba gurus!)  So if you are hooking all the IO 
calls, you'll need to do that yourself.  Fortunately, for my 
application, I hook the calls, but eventually pass control over to the 
default_vfs stuff.

However, again for my specific application, it turns out not to matter.  
The reason for this is that a PB class system is unlikely going to be 
used by interactive users; it is used by applications, and they 
generally don't care too much about the notifications.


More information about the samba mailing list