idl compilers and freedce threading / samba interop issues

[Luke Kenneth Casson Leighton]

On Wed, Jan 09, 2002 at 01:44:50PM -0800, Andrew Tridgell wrote:
> Wez,
> 
> > If *anyone* is considering work on IDL or NDR DCE/RPC code generation,
> > it *really* *only* makes sense to contribute that effort into the freedce
> > IDL compiler; as Luke says it is a *huge* project to build from scratch.
> 
> While I think that freedce is a very worthy effort, and I admire the
> work that has gone into it, I'm not completely sure it's the only or
> even the best way to do MSRPC on unix.
 
> The problem with the freedce code is that it's based on an extremely
> large implementation. There are much smaller IDL compilers out there
> and there are IDL compilers that generate tighter code. The H/Direct
> compiler (written in Haskell) is quite interesting and Flick shows how
> highly optimized code might be produced.
 
 well, the point about freedce is that it exists, and it works.

 it does an extremely good job, was written by the people who
 developed the specs, is extremely well documented.
 
 the known problems with it [you can crash it v. easily with
 rpctorture or just overloading it with half-baked requests!]
 can be catered for by adding in an exception handler at
 an appropriate point that traps the exceptions the runtime library quite deliberately throws, and/or by simply sitting down for a full month
 running rpctorture on a few choice applications and hammering
 away at it until it works.


> Last time I looked at the dcerpc code the main problem I had with it
> apart from its size was the tight linkage to a threads package. Maybe
> if I had put more time in I could have seen how to avoid the threads
> system, but at the time it certainly seemed to be pretty deeply
> embedded in the runtime system and the generated code seemed to be
> pretty closely tied to the runtime (due to the interpreted table
> approach to IDL compilation).

 yes.  the reason for this is that threading is quite important
 to implement, given that clients may wish to do threads, they
 had to provide a threaded implementation of client / server
 architecture.

 samba's spoolss implementation runs into quite awkward
 implementation difficulties due to not having threads:
 you need to _emulate_ threads by saving state information
 that would normally be communicated via in-memory data
 structures, you need to put them on-disk (tdb) or in
 shared memory.

 i'm referring, of course, to the spoolss change notify
 system.  the _client_ is multi-threaded: one thread for
 the change notify, the other thread for the actual
 printing.  that's why you get an RPC_ALTER_CONTEXT
 connection [it's for the [second] change notify thread].

 anyway.


 the server-side listener approach [taken by freedce] is pretty
 much the same as that used by apache, which is well-known.

 there is a pool of threads available.  there are a number of
 connections being maintained, with associated state information.

 the number of connections is allowed to significantly exceed
 the number of threads.

 any thread may be used to process an incoming request over any
 connection.

 clearly, when all threads block or take too long to process a
 request _then_ you run into difficulties.

 i haven't investigated what freedce does when this occurs:
 maybe it allocates another thread?

 
 however, as you say, it would be a hell of a lot of work
 to turn the dependence on the code into non-threaded.  i mean,
 it would be _unbelievably_ a significant amount of work.

 you would need to identify every single piece of memory
 (which is mutex locked / unlocked, so maybe it wouldn't
 be _that_ difficult) and data structure that is thread-safe,
 and set up a shared memory wrapper around it instead.

 basically, it would be a god-awful job.


> Having to use a particular threads package (or any threads package at
> all) makes freedce a portability nightmare, and also raises really
> nasty problems with security contexts (in general OSes don't offer
> per-thread security contexts).

that _is_ okay.  dce/rpc applications are expected to do their
own security context implementations - in userspace, which is
really the best place to do it _anyway_, for the majority of
applications.

there is one exception to that, of course: filesystems and
atomic file access where the file access is expected to conform
to _unix_ file system behaviour.

where access to the file system is _not_ expected to be done
via any interface _other_ than the dce/rpc one, then there is
no problem here.

so, a dce/rpc file system such as DFS is okay, including when
you do a kernel-level client.

it's when you store the files used by the DFS server in a unix
filesystem and then bypass the DFS server that you are guaranteed
to get atomicity and race condition issues hitting you.

[i listened very carefully to what you were saying, on file
systems, andrew :) ]


> For a while now I've been aiming to produce a small IDL compiler that
> is appropriate for code generation for Samba while being complete
> enough for all of the IDL code that NT uses. As Luke mentioned my
> first attempt was 'aparser' which is in the Samba source tree. It was
> written in awk, and while it was fun to do the structure means that
> it really doesn't have a chance of being a serious compiler.
 
 yeh: it kinda generated code that was compatible with the hand-
 generated marshalling / unmarshalling code i pioneered.

 given that none of us except matty understood at the time what
 the difference between an encapsulated and a non-encapsulated
 union, i think it did a pretty damn good job!

> My 2nd attempt is called pidl (see the cvs module by that name on
> samba.org) and I am much more hopeful about this one. I hope that this
> year we will see large parts of the MSRPC code in Samba replaced by
> auto-generated code from pidl. 

 coool!

> pidl uses a proper parser generator and
> grammmer description which seems to cover enough of IDL for the
> current NT calls, and has a structure that I think will make it fairly
> easy to extend. The big advantage of pidl is it's size. It weighs in
> at about 1k lines of perl now and will probably reach 2 or 3k lines by
> the time in can do really useful work. 

 have you looked at the BNF idl definitions we put together?

 i think it was... sander? matty?  david flynn?  who worked it out.

 it was a long time ago.

> Of course it may be that freedce really is the future of MSRPC on
> unix, but I wouldn't want to completely dismiss other options at this
> stage.

 no, and i don't think that anyone should.

 heck, i mean, IBM has their 3.0 DCE/RPC stuff [which is dce 1.2.2
 with bug-fixes and about an extra 1 million lines of code in
 services, transport plugins and security plugin modules] which
 people could use.

 then, god forbid, there's even MIDL.exe if people compile samba
 under cygwin!  [yes, rpcclient compiles and works under cygwin!]

 lkcl

> Cheers, Tridge
> 
>