MIDLC IDL Compiler

[Andrew Tridgell]

Mike,

 > Huh? They have conformance information in the header. You don't need
 > the zero terminator.

It is common for the Microsoft server code to check the length to see
if it exactly matches what they expect. So, for example, lsa mostly
uses a non-terminated string. netlogon mostly uses terminated
strings. Microsoft cope with this by forcing people who program each
interface to know which convention is used. For example, see this:

 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/secmgmt/security/using_lsa_unicode_strings.asp

That is incredibly error prone. We cope with this using a flag
extension on strings to indicate the termination. We also check
incoming strings to see if they follow the indicated convention, and
if they don't then we give a warning, as it might indicate our IDL is
wrong (we've found quite a few bugs this way).

 > Instead of speaking about vague concepts can you provide a specific
 > example of a "vital" extension? Please tell me what call or provide an
 > IDL fragment that illustrates clearly that only a compiler extension
 > could support reasonably.

A good example is the string handling above. Others include the
relative pointers needed for spoolss, and for coping with security
descriptors sanely. We also have extensions for forcing little-endian
(needed to encode epmapper.idl and for parts of DCOM) and for forcing
no alignment (needed for epmapper and the krb5 PAC).

Then we have the "gensize" extension to generate XXX_size() functions,
so we can correctly fill in the size variables in drsuapi and
epmapper, without relying on sizeof() being right, which it won't be
on many architectures.

Then we have extensions to make life easier for programmers, like
value() for automatically filling in well-known values, and the
various print extensions for auto-generating debugging functions that
can print complex structures.

 > > where do you get these claims about pidl from? We handle (and
 > > validate) the conformance value. Look for NDR_ERR_CONFORMANT_SIZE in
 > > our generated code. It's all over the place.
 > 
 > No. I'm not talking about a regular array.

oh for heavens sake, will you please stop assuming I'm a complete
moron? I know what a conformant array is, and we have implemented them
in pidl for ages. Look for "conformant" in the generated code or in
the compiler. Or look for them in the IDL in lsa.idl, mgmt.idl and
svcctl.idl. lsa_PrivilegeSet is a good example.

I didn't use a conformant array for dom_sid2 as I wanted it to be type
compatible with the ones that can't be represented in that way.

 > Can you please provide examples of two calls that illustrate the different
 > encodings?

In a security_ace and a security_descriptor they don't use a
conformant array in the wire format (and in fact they don't even
encode it as IDL). So, to allow us to use the same C type in both
places I used a cpp hack. It generates exactly the same wire format,
and does the same checks as if I had used a conformant array.

 > Could you imagine me trying to get PIDL to do Java? Seriously.

I didn't propose you use pidl, but I proposed you at least read it so
you don't make the same mistakes we made.

 > Why didn't you use MIDL syntax for your IDL? You don't need to change
 > the *syntax* to implement your extensions.

We try to support the MIDL syntax as much as possible, and we continue
to try to support more, but we also support syntax that MIDL doesn't
when that makes sense.

 >   o You use [out] only string parameters. Illegal in MIDL.

yes we use those at the moment. I'm not completely wedded to them, and
we could recode our IDL to use a different syntax, as long as the
resulting application code doesn't become too cumbersome.

 >   o You don't support embedded conformant arrays (e.g. dom_sid2 vs.
 > dom_sid).

yes, we do support them

 >   o You don't support [string] so making people convert to unistr is
 > "easyier"?

yes, I think "unistr" is vastly easier to deal with, for the reasons I
have explained. I don't mind at all adding support for [string], but
we haven't yet hit a need for it.

 >   o Pointer parameters are not [ref] by default.

we support the midl syntax for [ref] pointers. We also support an
alternative syntax that can be neater in some cases.

 >   o No import (this would be problem for Java because it needs to know
 >     the interface from which the symbol came).

We use depends() for this, but its certainly a possibility to add
support for the MIDL import facility if that can be made to work
reasonably. I haven't tried, so I don't know how hard that would be.

 >   o You've got custom attributes like [value] that does who knows
 >   what.

it allows you to embed arbitrary expressions in size_is() and
length_is() attributes. This is useful as we need to use external
functions to calculate things for size_is() where Microsoft doesn't
have to. A good example is where you have this:

	typedef struct {
		[value(2*strlen_m(r->string))] uint16 length;
		[value(r->length)] uint16 size;
		unistr_noterm *string;
	} lsa_String;

as we store the string as a "char *", we need some way to fill in the
"length" field correctly. This is a lot more complex for Unix strings
that it is for win32 as we deal with arbitrary multi-byte strings. So
the "char *" string could be UTF8, with between 1 and 3 bytes per
character, or it could be shift-jis, with mostly 2 bytes per
character. If we forced the application code to have to know the
details of this then it would drive developers mad. By getting it
right in one place (in the IDL) we make our code much less error
prone.

The syntax for value() is quite horrible, but the concept is fine. We
should support arbitrary expression parsing in value() so we don't
need the ghastly r-> stuff, and instead treat it as just another
arbitrary IDL expression. Someday that will be fixed and the value()
stuff will get cleaner. Meanwhile, the code that uses these structures
doesn't get horribly complicated by having to deal with it whenever
the structure is used.

 > Why? What advantage does it give you to do this stuff?

see the above examples, or read more of our IDL