[ceph-users] CTDB Cluster Samba on Cephfs

[Jeremy Allison]

On Wed, Apr 03, 2013 at 06:26:05PM -0400, Simo wrote:
> On 04/03/2013 05:42 PM, Jeremy Allison wrote:
> >On Wed, Apr 03, 2013 at 03:53:58PM -0500, Sam Lang wrote:
> >>On Thu, Mar 28, 2013 at 6:32 AM, Kai Blin <kai at samba.org> wrote:
> >>>-----BEGIN PGP SIGNED MESSAGE-----
> >>>Hash: SHA1
> >>>
> >>>On 2013-03-28 09:16, Volker Lendecke wrote:
> >>>>On Wed, Mar 27, 2013 at 10:43:36PM -0700, Matthieu Patou wrote:
> >>>>>On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> >>>>>>Hi list, I'm trying to create a active/active Samba cluster on
> >>>>>>top of Cephfs I would ask if Ceph fully supports CTDB at this
> >>>>>>time.
> >>>>>If I'm not wrong Ceph (even CephFS) do not support exporting a
> >>>>>block device or mounting the same FS more than once whereas CTDB
> >>>>>explicitly require that you have a distributed filesystem where
> >>>>>the same filesystem is mounted across all the nodes.
> >>>>Is that true? I thought Ceph was one of the cluster filesystems
> >>>>doing just that. What is Ceph if not a cluster file system?
> >>>There's some problem with mounting the in-kernel cephfs driver on
> >>>systems running the osd, iirc. I had to use the fuse-based driver to
> >>>mount, which obviously is not too great, speed-wise.
> >>>See http://ceph.com/docs/master/faq/#try-ceph for a better description
> >>>of the issue.
> >>Just to let folks know, we have a ceph vfs driver for samba that we
> >>are testing out now.  We're planning to resolve a few of the bugs that
> >>we're seeing presently with smbtorture, and send a pull request to the
> >>samba repo.  If anyone wants to help with testing, let us know.  The
> >>changes currently reside in the ceph branch of
> >>http://github.com/ceph/samba.
> >One thing I noticed. Returning a struct ceph_dir_result *
> >cast to a DIR * struct won't work in 4.0.x as we expect
> >dirfd(DIR *) to return a valid file descriptor.
> This sounds like a bug in 4.0.x, we should add a VFS layer operation
> for dirfd if we do not have it already.

Ok, figured this out. Here's what I'm going to do.

It's possible to remove all uses of dirfd(DIR *) completely
from the Samba code (and someone started using it in one
of the modules where it's not needed, so I can remove it
from there also). It was my mistake to add it, it's not
needed to do what needs doing in source3/smbd/dir.c and
I can achieve the same thing merely by having an additional
back-pointer to the owning fsp in the internal dir struct.

So for 4.0.x this will suffice to fix the bugs caused
by not treating DIR * as a competely opaque pointer
by using it in dirfd(). I have most of that patch
written already, but I'm off tomorrow so will finish
it up, log a bug and attach the fix to it on Monday.

For master - aka 4.1.0, I'll change the VFS opendir
and fdopendir operations to return an different opaque
pointer type such as SMB_DIR * (interestingly, this
is what we originally used to do inside the VFS) which
is deliberately defined as a pointer to an undefined
struct (or a void).

I don't think we need then to add a new dirfd() VFS
op, as there will be no internal code that needs it,
and making the Samba VFS opendir()/fdopendir ops return
a differnt pointer type than DIR * should prevent people
from assuming that can just call dirfd() on that pointer
type. If someone comes up with a compelling need for it
later we can always revisit and add it before 4.1.0
ships (hopefully by the end of this year).

Cheers and have a good weekend everyone !

Jeremy.