[ceph-users] CTDB Cluster Samba on Cephfs

[Volker Lendecke]

On Fri, Mar 29, 2013 at 11:05:56AM -0700, ronnie sahlberg wrote:
> On Fri, Mar 29, 2013 at 9:31 AM, Marco Aroldi <marco.aroldi at gmail.com> wrote:
> > Still trying with no success:
> >
> > Sage and Ronnie:
> > I've tried the ping_pong tool, even with "locking=no" in my smb.conf
> > (no differences)
> >
> > # ping_pong /mnt/ceph/samba-cluster/test 3
> > I have about 180 locks/second
> 
> That is very slow.
> 
> > If I start the same command from the other node, the tools stops
> > completely. 0 locks/second
> 
> Looks like fcntl() locking doenst work that well.
> 
> 
> 
> The slow rate of fcntl() lock  will impact samba.
> By default, for almost all file i/o samba will need to do at least one
> fcntl(G_GETLK) in ordet to check whether some other, non-samba,
> process holds a lock to the file.
> If you can only do 180 fcntl(F_*LK)  per second across the cluster for
> a file (I assume this is per file limitation)
> this will have the effect of you only being able to do 180 i/o per
> second to a file, which will make CIFS impossibly slow for any real
> use.
> This was all from a single node as well  so no inter-node contention!
> 
> 
> So here you probably want to use "locking = no" in samba.  But beware,
>   locking=no can have catastrophic effects on your data.
> But without "locking = no"  would just become impossibly slow,
> probably uselessly slow.

Please make that "posix locking = no", not "locking = no".

The locking=yes piece is being taken care of by Samba and
ctdb.

> Using "locking = no" in samba does mean though that you no longer have
> any locking coherency across protocols.

s/locking/posix locking/

> I.e.  NFS clients and samba clients are now disjoint since they can no
> longer see eachothers locks.

Right. But that's kindof okay if you have only SMB clients.

> If you only ever access the data via CIFS,  locking = no  should be safe.

Again: Please DO NOT USE locking=no, the options is "posix
locking = no"!!!

> But IF you access data via NFS or other NS protocols,   breaking lock
> coherency across protocols like this could lead to dataloss
> depending on the i/o patterns.
> 
> 
> I would recommend only using   locking = no   if you can guarantee

"posix locking = no" please, not "locking=no". I know that
some product's command line interface calls this "locking",
but I doubt this is what is being used here.

> that you will never export the data via other means than CIFS.
> If you can not guarantee that,  you will have to reseach the use
> patterns very carefully to determine whether locking = no is safe or

And again: "posix locking = no", not "locking = no"

> not.
> 
> 
> 
> I think for fcntl() locking,   depending on use case , is this a home
> server where you can accept very poor performance?   or is this a
> server for a small workgroup?
> If the latter, if using locking = yes   you probably want your

Ok, we have a misunderstanding throughout this complete
mail. Again, I think you want "posix locking", not "locking"
set to no.

> filesystem to allow  >10.000 operations per second from a node with no
> contention
> and >1000 operations per node per second when there is contention across nodes.
> 
> If it is a big server, you probably want >> instead of > for these
> numbers. At least.
> 
> 
> But first you would need to get ping_pong working reliably  both
> running in a steady state, and later running and recovering from
> continous single node reboots.
> It seems ping pong is not working really well for you at all at this
> stage,   so that is likely a problem.
> 
> 
> 
> As I said,  very few cluster filesystems have fcntl() locking that is
> not completely broken.
> 
> 
> 
> 
> For now,   you could try "lokcing = no" in samba with the caveats above,

Please make that again "posix locking = no", not "locking = no".

I hope this message got through by now.

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de