[ceph-users] CTDB Cluster Samba on Cephfs

[Sage Weil]

On Wed, 27 Mar 2013, Matthieu Patou wrote:
> On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> > Hi list,
> > I'm trying to create a active/active Samba cluster on top of Cephfs
> > I would ask if Ceph fully supports CTDB at this time.
>
> If I'm not wrong Ceph (even CephFS) do not support exporting a block device or
> mounting the same FS more than once whereas CTDB explicitly require that you
> have a distributed filesystem where the same filesystem is mounted across all
> the nodes.

Er, sort of.

RBD presents a block-based interface.  As many clients as you want can use 
that at the same time, although the client caching should be disabled if 
the users don't call flush() to make their writes visible to others.  
Beyond that, RBD itself doesn't care how many people use it.  *However*, 
if you put a "local file system" on top of RBD (like ext4, xfs, btrfs, 
zfs, ...), only one client should use RBD at a time because those file 
systems are designed for exclusive access to the disk.  If you use a 
"clustered file system" like ocfs2 or gfs[2], multiple clients can share 
the same RBD volume in a useful way, but they expect coherency to behave 
like on a SAN (which means writes are visible immediately but not durable 
until flush), which means RBD caching should be turned off.

CephFS is designed for concurrent, shared, coherent access from many many 
clients.  Think NFS, but scalable, and coherent/consistent when clients 
are accessing the same files and directories.

As for CTDB: we haven't looked at this specifically in the context for 
Samba.  Generally speaking, anything you can do with NFS or another shared 
file system you can do with CephFS.  IIRC last time I discussed this with 
the Samba guys, there is more we could do here to make CTDB much more 
efficient (by backing it with RADOS, for example), but we haven't looked 
at any of this carefully.  We would love to see clustered Samba working 
well on CephFS, though!  If you haven't already, plese look at our 
samba.git cloen on github, which has patches gluing libcephfs directly 
into Samba's VFS, allowing you to directly reexport CephFS via Samba 
without a local mountpoint in the middle.

sage