MIDLC IDL Compiler

[Michael B Allen]

Andrew Tridgell said:
> The librpc/ndr/ layer in Samba4 does use a linear buffer to marshall
> into, but it expands this buffer as it goes along, using
> talloc_realloc(). See the function ndr_push_expand(). So the buffer
> grows as needed.

Ahh. Good. You had me worried for a minute that you were doing some odd
incremental decoding.

Actually about "pre-allocating the maximum possible buffer" I no longer
see what the big deal is. I mean if you're ultimately going pull all the
data into one full size buffer are you really going to save a lot of clock
cycles by decoding it in peices? Why not just wait until you have all the
data and then run your marshalling routine once? Are you decoding/encoding
when you would otherwise be blocked? Even if you are I wouldn't be
surprised if decoding/encoding in one step was actually *faster* because
of CPU cache locality/misses.

> Similarly, when unmarshalling, the ndr layer generates talloc
> allocated structures at each level.

Sure. Actually it should be no surprise that I'm planning on using my own
garbage collection here. I can't be dragging around talloc. I have the
luxury of being a little more explicit anyway. I can't be completely
explicit and free objects in the encoder routines because for whatever
reason the call may never make it that far. But it's trivial to keep a
simple stack of pointers associated with the ndr context. When the call is
complete (possibly do to an error that aborts the call in the middle) I
just free the objects on the stack with the ndr context.

> You may also find that you like some of the extensions we have made to
> IDL to make building IDL based code much easier.

Man I'm just trying to get echo.idl going here. Don't try to sell me
extensions :->

> This is even more important for string handling. It would be terrible
> for Samba to have to deal directly with the utf16 arrays that
> Microsoft puts on the wire. By using a helper function
> ndr_pull_string() we can let the application code deal with it as a
> native unix string, and have the helper function do all the dirty work
> of format conversion.

Absolutely. MIDLC passes anything marked as [string] to the following
functions:

int
enc_ndr_string(struct ndr *ndr,
        unsigned char *str,
        unsigned char **dst,
        unsigned char *dlim)
{
    unsigned char *start = *dst, *hdr = *dst;
    uint32_t size;

    if (dst == NULL ||
                *dst < (unsigned char *)0xFF ||
                (*dst + 12) > dlim) {
        return -1;
    }

    *dst += 12; /* skip header momentarily */

    if (enc_mbsncpy((char *)str, 0x7FFF,
                (char **)dst, dlim - *dst,
                -1, "UCS-2LE") == -1) {
        return -1;
    }
    size = *dst - start;
    enc_ndr_long(ndr, size, &hdr, dlim);
    enc_ndr_long(ndr, 0, &hdr, dlim);
    enc_ndr_long(ndr, size, &hdr, dlim);

    return *dst - start;
}
int
dec_ndr_string(struct ndr *ndr,
            unsigned char **str,
            unsigned char **src,
            unsigned char *slim)
{
    uint32_t size;
    int n;

    if (src == NULL ||
                *src < (unsigned char *)0xFF ||
                (*src + 12) > slim) {
        return -1;
    }

    *src += 8;
    dec_ndr_long(ndr, &size, src, slim);
    size *= 2;

    if ((n = dec_mbsncpy((char **)src, size,
                NULL, 0x7FFF,
                -1, "UCS-2LE")) == -1 ||
            (*str = malloc(size)) == NULL ||
            (n = dec_mbsncpy((char **)src, size,
                (char *)*str, 0x7FFF,
                -1, "UCS-2LE")) == -1) {
        free(*str);
        return -1;
    }

    return n; /* number of bytes in new string not including zterm */
}

But I accept that *this* will need to be parameterized as you're obviously
not going to want to drag around my custom libiconv string conversion
interface. You're going to want to use your own string routines.
Ultimately I think the protos are going to be something like:

int dec_ndr_string(struct ndr *ndr, iobuf_t *buf);

but it would be pretty easy to extend the [string] attribute to be like
[string(TYPE)] and then pass TYPE in like:

int
dec_ndr_string(struct ndr *ndr,
            int type,
            iobuf_t *buf);

That would be pretty flexible with all the different string types.

> You proposed reusing the buffer. As soon as you reuse marshalling
> buffers then you have to deal with the time the buffer becomes
> available.

Well I didn't really mean that you should use a *single* buffer for *all*
I/O. I just meant you could reuse buffers. You could have a pool. Or
something.

Mike