NamedPipe API design; freedce / samba interoperability issues

[Luke Kenneth Casson Leighton]

On Wed, Jan 09, 2002 at 12:16:26AM +1100, Martin Pool wrote:
> On  8 Jan 2002, Luke Kenneth Casson Leighton <lkcl at samba-tng.org> wrote:
> 
> > if not, versioning / support (like 2-stage SMB CAPABILITIES
> > negotiation) could be added, which makes it a future-proof interface.
> 
> The situation I was imaging is that if every release for the first
> year or so has bugs or unimplemented features it might be too hard for
> the hypothetical Exchange client to use, because they'd rather have
> their own bugs.  Therefore it's no good for them to use.  
 
 what is being implemented here is an API that a plug-in module
 (libdcencacn_np.so) inside of freedce will use [yes, freedce's
 transport API has been turned into a loadable module architecture,
 courtesy of wez.  it's very cool, and it was very easy for him
 to do, thanks to the OSF people's incredibly good design].

 both the API and the loadable transport module will be totally
 hidden from the hypothetical exchange client _and_ the hypothetical
 exchange server.

 all that these two will need to know is that they can use the
 string "ncacn_np" as well as the "ncacn_ip_tcp" in their app,
 and that they will need to call an rpc_winnt_set_auth_info()
 and rpc_winnt_get_auth_info() depending on whether they're
 client or server.

 these will set or get the NT user-security context, as appropriate.

 so it's an internal API that is not exposed at the application
 layer _anyway_, so the risk you describe is not only different
 from what you expected, but also minimal.


> If the only viable customer for the code is Samba, you might as well
> make it Samba-specific.

well ... i am not sure what you mean, here.

of course it's samba-specific, however the information
that i am recommending is transferred is the essentials,
and is even _required_ by freedce, as it happens
(the info obtained by samba's get_client_name() and
get_client_addr() functions are needed in the freedce
transport API to answer its equivalent function of
get_client_name() and get_client_addr() - funnily enough!)

you also _need_ the netbios name to be transferred,
because there is _another_ freedce api - rpc_get_princ_name()
iirc correctly - that needs this!

and then the NT user-security context, i hope that that
is obvious enough, it's the whole point of the Win32 NamedPipe
layer.

so yes, it is, and has to be, samba-specific, as samba is
the only viable "customer" because it's the only practical
open source SMB implementation available today.

if there were any other open source SMB implementations, i'd
be contacting them, too, you can count on that.

> Sure, code can be generated from IDL using any existing IDL compiler
> and support library.  But from what I understand there is no complete,
> free, suitable, well accepted such program.  I'm questioning whether,
> if we worked on one, anyone else would use it.

well, there is one that is sufficient for the purposes needed,
namely to generate hard-coded code that can parse a data structure
in IDL format.

it's called aparser, and andrew worked on it with a couple of
pointers and a bit of encouragement from me and matthew chapman.

it's very good code (for awk! :), is template-based so can be
adapted easily.

and, what is more, it will be able to generate code that can
be "slotted in" without modifications into the SAMBA 3.0 codebase.

the templates it uses to generate the code were written specifically
for the RPC code that SAMBA 3.0 uses.


> RPC libraries tend to make assumptions about how they will be used.
> For example, most that I've seen either make threads optional, or not
> supported at all.  

okay, i should make it clear that you're expecting to have to
use _way_ more than is actually needed.

for this pipe_glue API, all that is needed is the "pickling"
code [i know you know what picking is, you pointed out that
python has pickling support to me :)], which in freedce is
available as an independent API, and in SAMBA, you need the
aparser program.

the transfer of the data [NB names + TCP info + NET_USERINFO_3]
can be "pickled" by code generated by aparser, shoved down
a unix domain socket or shoved into shared memory, or even
files, mmaped or not, and then "unpickled" by freedce using
the unpickling API.

there is in fact a dce/rpc RFC that describes how to do
OSF pickling, if you're interested.

take a look at http://dcerpc.net/url, search the page
for "pickling" and/or for "RFC", then jump to the
frame-referenced page listed on the url annotation.

when i get home [and am not paying for mobile phone calls!]
i'll look it up for you.

> (Isn't freedce threaded, and Samba explicitly not
> so?)  

it is, but for the purposes of this discussion, this is irrelevant
as it is an implementation-specific consideration.

> Most their own method for handling exceptions, allocating
> memory, and so on.  If you already have a big chunk of application
> code, then there's a good chance it will be incompatible with the
> assumptions of the RPC library.
 
 okay.

 the whole point of dce/rpc is that it is transport-independent,
 network-independent, platform-independent etc etc.

 so as long as you can communicate "data" over the well-defined
 interfaces, even between two totally separate packages, one
 called SAMBA and the other called freedce (or an app that uses
 freedce), you're home and dry.

 that's what i'm recommending be leveraged, here.


 so by using "pickling", you can transfer NT security and
 NetBIOS and TCP socket information between one totally
 independent program and another, with no hassles.


> In my experience many people who want to use CORBA have to write a
> glue layer to cope with these issues.  GNOME tried out two or three
> ORBs (Mico etc) and ended up writing their own (ORBit) because the
> existing free implementation didn't fit with the kind of program they
> wanted to write.

i've done an "experimental" version of the NamedPipes API,
removing the NT security context from Samba TNG
and not putting NT sec-ctx code into a freedce-compiled
samrd SAM server.

it worked first time, and i came across no such problems.

if i get you correctly, then yes, i will need to write
a better ncacn_np transport layer, but that's because
the people who wrote the ncacn_ip_tcp and ncadg_ip_udp
transports used sendrecv() / BSD socket code etc., and
it has some difficulties when presented with unix domain
sockets, as i've found out :)

so i will have to rewrite it a bit, but it will be a
dynamically loaded, interchangeable and replaceable module
that will have _zero_ impact on the application developer
who uses freedce and freedce environment.


> > >  - It might not be possible to write a clean interface to dcerpc,
> > >    because of cross-layer coupling - e.g. authentication carried out
> > >    at different levels of the stack, etc.
> > 
> > that's implementation specific.
> 
> Well, we're talking about implementations here.
> 
> > it's in the idllib/ directory.
> 
> This is
> 
>   cvs -d :pserver:anoncvs at cvs.dcerpc.net:/cvsroot get freedce

.../cvsroot co freedce is what i know about.

> right?
 
  correct.

  you'll also need dcethreads, which is a "wrapper" api
  around the standard posix thread library that gives 
  posix draft 4 semantics instead of the semantics that
  posix later got changed to!

  ... but it's _still_ the same posix thread library 
  api [plus or minus a couple of arguments and functions *grin*]


> Is the documentation correct in asserting that it is all either MIT or
> GPL licensed?
 
 that is correct, ish.  it's actually the OSF 1.1? (1.0?) license,
 which has been certified as an open source BSD-compatible license.

 the dcethreads library is all GPL.

> FreeDCE being so large is an argument for not rewriting it on a whim.

 that is _definitely_ correct.

> On the other hand, since Samba in total is smaller than freedce and

 no it's most definitely not.

 freedce is 250,000 lines of code.

 SAMBA 3 was 340,000 lines of code, last time i checked, 2 years
 ago.  it's probably more like 380,000 by now, i guess.

 there is at least 80,000 lines of hand-crafted dce/rpc pickling/
 unpickling code is SAMBA, which freedce is capable of auto-generating.

 so it's a bit like comparing apples and pears when it comes to
 it, really.

 basically, each project is not insignificant, and has areas
 of interdependence and expertise.


> therefore obviously only uses a small fraction of the functionality,
> it seems silly to add a dependency on such a big chunk.  

 well, by using freedce, SAMBA can reduce its codebase by about
 100 _thousand_ lines of code.

 but that's not for me to say, and i'm not advocating it, either.

 as long as that pipe_glue API is in place, is part of a samba
 3.0 distribution real soon, and can be enabled at runtime,
 then you [samba team] don't need to hear from me _ever again_.


> I'm a little afraid that if we have to modify the RPC layer to
> accomodate M$ wierdness, 

 no, you won't.  in fact, you'll be able to take _out_ all of
 the RPC layer _all together_!

 you will find that you can link smbd the binary with NO reference
 to RPC code _whatsoever_, once you correctly implement a suitable
 dce/rpc "glue" loadable module interface.


> or evolution since the OSF version was
> written, 

there's actually very little, and myself and wez have already
found and made the necessary modifications sufficient to
compile exchange-compatible IDL file and NT-domain-service
compatible IDL files such as samr.idl and lsarpc.idl.

we've had sizeof(size_parameter / 2) for about six months.

wez was preparing the groundwork to do a more "generic"
maths function "thing", so that he could do this:

   sizeof((size_param | 0x7) & ~0x7)

which he tells me he needed for DCOM.


> the code would be so big and complex that it might be harder
> than starting from scratch and implementing the necessary subset.
 
 well, the work's already been done, and demonstrated as working,
 so that's a moot point, and not actually the case.

 it was in fact quite simple to implement the modifications
 to make freedce's IDL compiler MS-IDL compatible and to make
 the freedce runtime library actually support the MS-IDL
 changes.


> Here's something to ponder:
> 
>   http://www.joelonsoftware.com/articles/fog0000000007.html

i'll take a look at it when i get home.

> --
> Martin
> 
>