MIDLC: The MIDL Compatible IDL Compiler

[tridge at samba.org]

Mike,

 > Currently the C stub emitter isn't there but I'll look at it ASAP.

The emitter is arguably the biggest part of the task I think. It seems
simple at first, but there are a lot of details to get right. Lots of
IDL compiler projects have got to the "can parse IDL into a parse
tree" stage, but then get stuck on the emitter :-)

 > The only thing holding me back was your lack of interest.

my apologies for that. I did look at your compiler a while ago, but as
it was at the "can parse IDL" stage at the time I didn't really pay
much attention. There are at least 3 other IDL compiler projects for
Samba that got to that stage and got stuck (aparser, sidlc and fidl).

You've obviously got a lot further than any of those projects ever
did, so maybe you'll find the last bits easy too :-)

 > Weeell, I don't know much about PIDL extensions so I'll leave those to
 > you.

I think it might be difficult to just add them later, as they really
pervade the core of the code. Then again, I'd be glad if you show me
I'm wrong :-)

 > int
 > lsarpc_LsarLookupSids(void *context,
 >             const unsigned char *src,
 >             const unsigned char *slim,
 >             unsigned char *dst,
 >             unsigned char *dlim)

I know it's early days for MIDLC, but I hope you don't mind if I
provide some critique of the interface.

In the above, it really needs to use DATA_BLOB or similar, as you need
range checking. Please, you need to pass in information on the byte
order. 

The equivalent PIDL fn is this one:

  NTSTATUS dcerpc_lsa_LookupSids(struct dcerpc_pipe *p, 
                                 TALLOC_CTX *mem_ctx, 
                                 struct lsa_LookupSids *r)

the 'struct dcerpc_pipe' gives all the context info, the mem_ctx gives
an allocation context, and the 'struct lsa_LookupSids' defines the
parameters of the call.

Another thing to consider is async calls. PIDL generates a separate
_send() function for every call, which allows just one side of the
call to be made. This is essential for generating an async-enabled
API.


 >     params.retval = LsarLookupSids(context,
 >             params.handle,
 >             params.sids,
 >             params.domains,
 >             params.names,
 >             params.level,
 >             params.count);

this is the big problem. The LsarLookupSids() call in the above takes
one C argument per parameter. That means that every RPC call takes a
different number of parameters. That makes it impossible to use
function pointers and generic driver code.

For example, see dcerpc_ndr_request_send(). That function is the core
send routine for all our RPC calls, and even though it does all the
marshalling it has no knowledge of the IDL formats. It just calls
ndr->ndr_push() to do the hard work. That is possible because
ndr_push() has a prototype that matches all of the generated NDR
routines. 

Also notice that this is what makes the 'validate' code in Samba4
possible. If you look inside dcerpc_ndr_request_send() you will see
checks for the DCERPC_DEBUG_VALIDATE_IN flag, in which case all
marshalled data in unmarshalled then marshalled again and compared,
which allows us to runtime select the validation of our NDR code. That
would not be possible with MIDL.

I know that MIDL does it the way you have done it, but it really is a
very bad idea. We have gained a lot in the structure of Samba4 from
the simple decision to use a structure (in this case 'struct
lsa_LookupSids') rather than a MIDL like interface.

Would you be willing to consider changing this in MIDLC ?

 >     enc_ndr_long(ndr, obj->length, dst, dlim);
 >     enc_ndr_short(ndr, obj->impersonation_level, dst, dlim);
 >     enc_ndr_small(ndr, obj->context_mode, dst, dlim);
 >     enc_ndr_small(ndr, obj->effective_only, dst, dlim);

obviously error checking is needed, as we don't have java exceptions :-)

PIDL also generates a lot more than just the marshall/unmarshall
functions. It generates the following per IDL function:

 - the C structure header
 - the _push function (mashshalling)
 - the _pull function (unmarshalling)
 - the _print function (essential for debugging)
 - an optional _size function (essential for some IDL constructs)
 - an entry in the interfaces function table
 - the client stub
 - an async _send client stub
 - the server stub

Cheers, Tridge