first swipe at ctdb spec file

[ronnie sahlberg]

There is currently no pCIFS   so I "took" the name. :-)
It is obvioulsy a play on the pNFS acronym which is incredibly hyped.



pCIFS in this context means a load-sharing cluster where multiple
nodes export the same data read-write to a cluster.


This is similar to pNFS, however pNFS is not really clustered, instead
(since it is much easier) pNFS provides scalability by having only one
node that exports tha share and processes all the metadata operations,
and then to provide "scalability" it provides a mapping between file
data and storage so that clients can do file i/o directly to storage
bypassing the nfs server completely.
And thus reducing the workload on the, single, NFS server, providing
higher scalability than if the singler NFS server also had to process
the data.

While it would be possible and likely a superior solution for pNFS to
provide real read-write clustering across multiple nas heads in a
cluster,   that is more difficult and not the path that the IETF
workgroup has taken.

One way of looking at the common pNFS is that is is really just a
single server  but it provides some additional extensions in order to
reduce the amount of data that is passed across the interconnect
between the nfs server and the storage backend.
NFS servers are often more limited by the bandwidth of the SAN
interconnect than CPU or memory in the nas head itself.
Thus it is primarily a solution to reduce SAN utilization and
saturation that occurs rather than providing a clustered service.


It would certainly be possible, academically speaking, to build a
pCIFS implementation similar to the pNFS path  with only one single
node that runs samba and then letting all clients do i/o based on some
mapping protocol that maps between files and extents on a storage
device.
Howver,   that is really just a "quick fix" to get slightly better
than single node performance on a nas head.


a true read-write cluster like our pCIFS   and a proper pNFS
implementation would be vastly superiour to the path currently taken
by pNFS workgroup, but also much harder to imlement.


Comparasions between our pCIFS and the most common pNFS approach :
 * higher scalability. we distribute also metadata operations across
the different nodes in the cluster.
 * higher availability, we do not have a single point of failure (no
single metadata server)
 * much higher performance.   There is a cost associated with a file
mapping protocol which is non-zero   and history has shown that for
these "pNFS" implementations (there are several, some of which have
been in production use for many many years).
In many implementations this "cost" of filemapping means that one only
gets any real benefit for the case where one only works with doing
sequential access to very large files.


please try it out.
it is pretty cool stuff.


On 6/4/07, Christopher R. Hertel <crh at ubiqx.mn.org> wrote:
> tridge at samba.org wrote:
> > Chris,
> >
> >  > Quick question:  is this the pCIFS that Peter Brahm started working on
> a few
> >  > years back or something different?
> >
> > I've no idea what Peter worked on.
>
> Have you had a chance to look at pNFS (which is in the NFSv4.1 spec.)?
>
> A few years ago, Peter proposed a complimentary pCIFS which would allow
> clients direct access to the block storage.  Metadata management would be
> handled via CIFS but the clients, once they held an appropriate lock, would
> do direct block I/O (via iSCSI, Fibre Channel, or other methods).
>
> I was speaking with folks from Panasas a couple of weeks ago.  Garth gave a
> pNFS presentation and afterwards we started talking about the flexibility of
> the Linux implementation of pNFS, and the possibility that the client-side
> code could also be used to support pCIFS functionality.  The Panasas folks
> sounded interested, but I think they'd want to know more about the
> possibility of follow-through from both Samba and the Linux CIFS VFS client.
>
> So back to my question:  When Ronnie talks about pCIFS which pCIFS is that
> and what is it intended to do?  Do we have a collision in the name space or
> is someone (not unusually) way ahead of me?
>
> >  > I'm interested in the API at present.
> >
> > The API is there, but the client half of it (which is what you would
> > use) is not yet cleanly separated from the server. I've started doing
> > that, but it will take some time yet.
> >
> > We'll end up with a libctdb library, which is what applications will
> > use. The code in common/ctdb_client.c is the basis for that library.
>
> I will take a look.
>
> Thanks.
>
> Chris -)-----
>
> --
> "Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
> Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
> jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
> ubiqx Team -- http://www.ubiqx.org/     -)-----   crh at ubiqx.mn.org
> OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh at ubiqx.org
>