Talloc and Tdb spin offs

[simo]

On Mon, 2007-09-10 at 11:17 +1000, tridge at samba.org wrote:
> Simo,
> 
> In some ways they already are spun off. talloc is being used by a
> number of external projects, it just needs more promotion and better
> packaging.

I know, but I mean is that we, ourselves should become users of talloc
and tdb using them as external libraries. That's a spin off.

> I also think that the the biggest thing that would help with wider
> talloc adoption would be to integrate it with a malloc compatible
> library, so that code that calls malloc/free could be mingled with
> code that uses talloc without error (so you can use any malloc pointer
> as a talloc pointer, and can use free() a talloc pointer).

I am not sure this can be done in a useful way if you mean doing
something like #define malloc my_talloc_malloc(...).
The only case where that would work, I guess, is when you recompile an
application from scratch with all libraries, or, if you can intercept
glibc's malloc/free through some kind of LD_PRELOAD trick. Otherwise any
time you link to a library at runtime things will just break. I just
experienced a bug like this with some code in bind.

> Combined
> with this we need a lower per-allocation overhead. Currently its 32
> bytes, which is too large for some projects. That would allow a
> project to start using talloc without a major reorganisation of their
> existing code, and would make it much easier to use in code that uses
> a lot of existing external libraries.

> Rusty and I have discussed some ways of doing this, and we think we
> can get the average overhead down a lot, while making it malloc
> compatible, by adopting some of the ideas from
> http://samba.org/~tridge/junkcode/alloc_mmap/ and integrating them
> into talloc.

I am reading this code and it seem interesting, but one thing that seem
a no-go, imo, is the "compile time" thread safety. If you use this code
in a library you want to be thread safe, but possibly only when
necessary. If it is a compile time option that means a
developer/packager will have to always compile in the thread safety
functions, just in case

I guess the use of mmap is to guarantee you have memory aligned to a
page boundary so that you can use the page base address to store the
metadata without having to prepend it to each memory allocation, but
always with an easy way to retrieve it.

Is there any other advantage?
Have you considered systems with huge pages ?

> Meanwhile, packaging talloc and pushing it into the distros would be
> very worthwhile. Any volunteers to do that? Jelmer has already done it
> for Debian (thanks Jelmer!). Would anyone else like to do it for RPM
> based distros?

I am willing to package them for Fedora.
I have 2 ways:
1) Keep them in samba and just create sub-packages of the samba SRPM.
2) Make them their own packages with their own spec files.

For the 2nd though I'd really like to have official and version numbered
releases, and possibly make our samba packages use them instead of
recompiling them again in the samba packages.

Thoughts?

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer
email: idra at samba.org
http://samba.org