Shared memory and IPC

[Jim McDonough]

>>> My advice is to avoid MAP_FIXED at all costs.
>> absolutely love to.  however, it's a speed improvement of
>> a potential two or maybe even three orders of magnitude,
>> and so cannot be ignored.
.....
>    Consider the following code snippit
>
>    struct {
>         int  this;
>         long      that;
>         char *theOther;
>         ...
>         char placeTheOtherPoints[MAXBUF];
>    } thingie;
>
>    base =  mmap(addr, len, PROT_READ|PROT_WRITE, MAP_SHARED, 42, 0)
>    ...
>    myThat = base->that;
>    ...
>    base->theOther = newOther - base;
>
>    The "that" value is just at an offset into the mmap'd
>    structure, so getting it's value is trivial: most
>    compilers turn this into a ",base,offset"
>
>    the Other value is trickier: it's a pointer to an address
>    in the mmapped area, so you have to add the base before
>    dereferencing it and subtract the base before updating it.
I think Dave's got a good idea here.  I've worked on a product that did
exactly this, and the overhead of adding the offsets was trivial (on AIX).
The product was originally on AIX only, where the first mmap is guaranteed
(if you're not using the Large Program model, which you control at build
time, so we had control) to be at 0x30000000.  We used pointers with the
addresses in that mmap'ed region.  Then, to port to other platforms, we had
to start using offsets instead.  The performance drop on AIX was not
measurable.  The data transfer rates we got between processes was huge
(basically was throttled by the process that put it out on the net saying
"stop, I can't take any more").  Yes, strictly speaking, there had to be
more overhead, but we just couldn't see it.

The only thing that helped was going from mmap() to shmat(), because we
didn't need anything to stick around in a file, and the mmap'ed file
changes were being flushed to disk every minute by the sync daemon, and
flushing 32 MB to disk doesn't happen instantaneously.  Going to shmat()
and backing it in page space prevented the sync and the hiccup every
minute.  But that's a different issue altogether.

----------------------------
Jim McDonough
IBM Linux Technology Center
6 Minuteman Drive
Scarborough, ME 04074
USA

jmcd at us.ibm.com

Phone: (207) 885-5565
IBM tie-line: 776-9984