[Samba] Custom VFS

[Jeremy Allison]

On Tue, Jul 24, 2012 at 02:35:28PM -0700, Andrew Scherpbier wrote:
> Hi Daniel,
> 
> Just a note of encouragement...
> I have so far written 2 filesystems in Java that use Samba for 2
> different companies, so you're not alone!  :-)
> 
> The strategy I've used is to write a simple TCP protocol client (the
> VFS module) and server (a straight forward threaded Java server).
> Works like a charm.  As long as the client side is abstracted enough
> so that its samba connection state is independent from the server
> connection state, there are no issues with restarting either.  (I
> started out using a statefull protocol, but ended up changing to a
> completely stateless one, where the individual messages contain
> enough information to establish context.  This way, if either end of
> the system goes down, recovery is the simple act of building a new
> TCP connection.)
> 
> I also attempted to use the Apache ActiveMQ C++ library for
> communication, but found it buggy and leaky.
> 
> I originally looked into hosting the JVM in the VFS module, but that
> was going to be a problem because each smbd process would have to
> start its own JVM.  The JVM startup time (especially the server JVM)
> is very high and the memory overhead would not make it scalable.
> 
> TCP through the loopback interface is very fast (at least on the
> linux system's I've developed for), so there was no need to
> implement some sort of shared memory interface.
> 
> The system I'm working on now manages PB class storage (currently up
> to 10PB) with hundreds of concurrent clients and the VFS module does
> this without issues or much overhead.  We're regularly seeing write
> speeds in the 400-500MB/s range using 10GbE and multiple windows
> clients.
> 
> Good luck!
> 
> P.S.:  Blatant plug for my current project:
> http://www.cuttedge.com/psca/index.html

Wow - that's really cool stuff !

I'm glad the VFS works so well for you. I wanted to give you
a heads-up on the changes we're making to the VFS moving
forward with 4.0.x and above - take a look at the changes
Volker made for the pread() -> pread_send_fn()/pread_recv_fn()
and pwrite() -> pwrite_send_fn()/pwrite_recv_fn() in order to
make the VFS async (and allow pthreaded implementations to
be hidden under the covers).

Sample implementations are in source3/modules/vfs_default.c
in:

vfswrap_pread_send()/vfswrap_asys_ssize_t_recv()
vfswrap_pwrite_send()/vfswrap_asys_ssize_t_recv()

It makes the VFS a little more complicated, but should
enable you to get more performance out of it.

We're also thinking longer term about changing the
model of keeping the current working directory as
the root of the exported service and changing the
internals of Samba to chdir() to the parent directory
of any path currently being processed - this allows
easier security checks inside smbd and reduces the
opportunity for pathname check race conditions.

Feedback very welcome - especially from someone
who has implemented a couple of production Samba
VFS modules already :-).

Thanks !

Jeremy.