[ceph-users] CTDB Cluster Samba on Cephfs

[Jeremy Allison]

On Thu, Mar 28, 2013 at 07:41:55AM -0700, Sage Weil wrote:
> On Wed, 27 Mar 2013, Matthieu Patou wrote:
> > On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> > > Hi list,
> > > I'm trying to create a active/active Samba cluster on top of Cephfs
> > > I would ask if Ceph fully supports CTDB at this time.
> >
> > If I'm not wrong Ceph (even CephFS) do not support exporting a block device or
> > mounting the same FS more than once whereas CTDB explicitly require that you
> > have a distributed filesystem where the same filesystem is mounted across all
> > the nodes.
> 
> Er, sort of.
> 
> RBD presents a block-based interface.  As many clients as you want can use 
> that at the same time, although the client caching should be disabled if 
> the users don't call flush() to make their writes visible to others.  
> Beyond that, RBD itself doesn't care how many people use it.  *However*, 
> if you put a "local file system" on top of RBD (like ext4, xfs, btrfs, 
> zfs, ...), only one client should use RBD at a time because those file 
> systems are designed for exclusive access to the disk.  If you use a 
> "clustered file system" like ocfs2 or gfs[2], multiple clients can share 
> the same RBD volume in a useful way, but they expect coherency to behave 
> like on a SAN (which means writes are visible immediately but not durable 
> until flush), which means RBD caching should be turned off.
> 
> CephFS is designed for concurrent, shared, coherent access from many many 
> clients.  Think NFS, but scalable, and coherent/consistent when clients 
> are accessing the same files and directories.
> 
> As for CTDB: we haven't looked at this specifically in the context for 
> Samba.  Generally speaking, anything you can do with NFS or another shared 
> file system you can do with CephFS.  IIRC last time I discussed this with 
> the Samba guys, there is more we could do here to make CTDB much more 
> efficient (by backing it with RADOS, for example), but we haven't looked 
> at any of this carefully.  We would love to see clustered Samba working 
> well on CephFS, though!  If you haven't already, plese look at our 
> samba.git cloen on github, which has patches gluing libcephfs directly 
> into Samba's VFS, allowing you to directly reexport CephFS via Samba 
> without a local mountpoint in the middle.

Oh, I didn't know about that ! Any chance of submitting it
to the main Samba repository so we can keep it up to date
with VFS changes ?

Jeremy.