hmm.. Re: talloc issues

[Sam Liddicott]

I hope I'm not patronising, but I'm evangelising to many readers here...

* Jeremy Allison wrote, On 30/07/09 16:32:
> On Thu, Jul 30, 2009 at 04:58:01PM +0930, Rusty Russell wrote:
>> On Wed, 29 Jul 2009 02:17:00 am Jeremy Allison wrote:
>>> +1. talloc_reference broke my brain a long time ago,
>>> and I don't want to maintain any code that contains
>>> it.
>> There are patterns which require it; the clearest to me is the "cache"
>> pattern where we want to keep an object around for later reuse, so we
>> hand out references.
>>
>> Here's a trivial example:
> 
> Yes, that's very clever, but IMHO it's an example of
> "too dangerous to use" design. What is going on with
> the memory references is not obvious to a casual user
> of the cache.

mmm... casual memory management...

- sure the user has to be introduced to reference counting if they want
to use a library that depends on reference counting, but that's not the
problem I think;

The problem is that everywhere people do reference counting (apart from
common subsystems like COM) there is a new way of doing it and users
always have to learn a new way. Remembering to call the right ref
counting API for a given struct type isn't a new problem with talloc
references.

Many libraries have their own free function that you have to use to free
data that the library allocates. None of this is ever obvious to a
casual user.

So I think that this contrived example can't be used to demonstrate the
danger as it was only contrived to demonstrate the use.

> If you really want references like this, you should
> be using a language that supports it by design. Yes,
> I know C++ sucks, but it does solve this kind of thing
> nicely :-).

Or.... if you want references like this you could use a malloc
replacement that supports it by design - like talloc** - but you only
have to use as much as you need.

[**Hopefully it will support it by design real soon now :-) ]

Here's the levels of talloc usage (#define's are for illustration, not a
style guide)

1. malloc replacement...
#define malloc(s) talloc_zero(NULL, (s))
#define free(s) talloc_free((s))

Level 1 has no advantages over malloc, but....

1a. ...with automatic free to make your code simpler
directly use talloc_zero(parent, ...
instead of "malloc"

That's samba3

2. reference counting (which you don't have to use)
#define malloc(s) talloc_zero(NULL, (s))
#define ref_inc(s) talloc_reference(NULL, (s))
#define ref_dec(s) talloc_unlink(NULL, (s))
#define free(s) talloc_free((s))

and suddenly you can reference count all your pointers - even char*,
with auto-freeing on zero refcount. Wow. Just like COM.

And you can still pass pointers to level 1 library calls that transfer
ownership (with talloc_steal or talloc_free).

2a. ...with automatic ref_dec where you want it:
directly use talloc_zero(parent, ...
and talloc_reference(parent, ...

Most API's that work at level 1 have a concept of passing-ownership,
which implicitly controls who is responsible for freeing a pointer.

Pure reference counting removes the need for this concept, but because
level 2 is a superset of level 1 the concept of passing ownership
remains. You don't have to write library functions that pass ownership,
but you can.

You can use talloc at whatever level you like and it won't** interfere
with usage at another level.

But (relating to the cache example) when using a library you'll still
always be tied to whatever memory management your library is using, and
talloc_reference/talloc_free is no worse than any other; e.g. the one
which gd offers.

> Talloc_reference is "too clever to use" C programming.
> Like multiple inheritance in C++, it's like a language
> feature that will trip everyone up sooner or later :-).

Level 2 is only too complicated when the problem is simple enough not to
need it - but then you stick to level 1 usage.

When the problem gets more complicated than reference counting... then
it's simpler to start reference counting.

..and talloc is the easiest way to reference count - you don't have to
change your structs for a start, and you don't have to port to C++ either.

And you can either "count" with NULL references, or do reference
tracking with automatic freeing. You can mix'n'match.

I think it is the one-size-fits-all allocator, which is why I want to
fit it into gnome, and swig, and other libraries. So that
talloc_reference() on a gtk widget will pass-through to gtk's refcount++
as well as keep a talloc_reference. This can work even if gnome uses
talloc in it's dumbest level 1 mode. (It means that each malloc pool has
an interface associated with it so that talloc knows which function to
pass-through the refcount++/-- calls to, etc).

And so, I hope, one day, good memory management won't require a port to C++

Sam

Sam