efficient rpc io using vstr

[Michael B Allen]

Martin Pool said:
> I think of Vstr as primarily a layer for allocating byte arrays, and
> secondarily a string library.  You can do things with its allocations
> that don't make sense in C strings: split them, delete parts, and have
> embedded \0s.

That's consistent with my understanding.

> Right, in all those cases you do need to copy.  Vstr does not address
> that particular part of the problem.  On the other hand, I think it
> will not hurt either, and might make it better.  For example, rather
> than convert_string_allocate, call
>
>   Vstr_base unix_vstr = convert_vstr(CH_UCS2, CH_UNIX, wire_vstr);
>
> The contents of that function would be similar to at present; rather
> than malloc it makes a new vstr.

I don't know Matin. I think vstr is going to malloc that.

> To fix up the path:
>
>   size_t pos = 1;
>   while ((pos = vstr_srch_chr_fwd(path_vstr, pos, path_vstr->len,
>                                   '\\')) != 0) {
>     vstr_sub_buf(path_vstr, pos, 1, "/", 1);
>   }

You can do this and canonicalize in one step with the function I posted.

> Speaking of canonicalizing paths: this is something that is
> persistently difficult to do in plain C strings without either
> overflowing or getting the wrong result.  There have been many holes
> in Apache, ftp daemons, IIS, and other programs because of trying to
> do it in low-level pointer arithmetic.

That canonicalization routine is an FSM, it handles every case[1], and it
never processes more than one character without checking the sentinel
pointers so it's at least not going to run off. Even if it has a hole it's
not big and would be easy to fix.

>> in 5 SMBs in one read you *might* be able to use vstr to manage multiple
>> smbs in the same buffer. But that could get really hairy.
>
> It's not even just throughput: if you want a nonblocking server then
> you need to be able to accept and hold partial packets, and similarly
> for output.  That implies some kind of buffering.  I think doing it
> through vstrs is pretty clean.  What did you think of the main() in
> that file?

I agree. Buffering might work out but I really think general string
processing is going to be a dud.

> and second [suba feature]?

All cells are size_t references from the start of the memory area so you
can copy, save, restore, or allocate from shared memory between mulitiple
processes, etc. This can be used to achive a variety of useful effects
such as transaction control, checkpoints, complex archives, etc.

>> int
>> myfn(void)
>> {
>>     unsigned char mem[0xFFF];
>>     struct allocator *suba = suba_init(mem, 0xFFF, 1, 0);
>>
>>     /* allocate memory */
>>     obj = suba_alloc(suba, 10, 1);
>>     /* and just return; no freeing necessary*/
>> }
>
> That's very clever.  On the other hand, there is no protection against
> overflow, which I think ought to be a major consideration for any
> network code.

Huh? How would this overflow? If you allocate all of the memory you get
ENOMEM.

Mike

[1]
http://www.ioplex.com/~miallen/libmba/dl/tcase/tests/data/PathCanonExamples.data