tevent_loop_set_nesting_hook deprecated?

[simo]

On Fri, 2009-03-20 at 10:26 +1100, tridge at samba.org wrote:
> Hi Simo,
> 
>  > Same feeling here.
>  > 
>  > Processing more than 1 event at a time is a recipe for madness
>  > timeout handling is not the only thing that go bonkers, think of an
>  > event that triggers a change in configuration,
> 
> processing more than 1 event at a time is called concurrency. That is
> what we are aiming for here. Yes, it's hard, and yes, it's hard to
> debug. 
> 
> Non-async code is easy to debug. That is what Samba3 uses, and its
> great, as long as you don't mind losing concurrency.

The problem is not just being hard to debug, I personally don't see
semi-async code much harder to debug then fully async, actually I don't
feel any difference.

> Fully-async code is very hard to debug and makes for extremely verbose
> code where the async handling dominates the code. The current ldb code
> is a good example of this. It is great to do fully async when it is
> worthwhile and high degrees of concurrency are expected.

The ldb code is not a very good example IMO (and yes I know I wrote it).
Tevent is a clear example of a better API.

> semi-async code is much simpler, as it allows the mainline code to
> continue in a non-async fashion. It can just call a routine (such as a
> winbind lookup) and wait for a result. It is definately harder to
> debug than non-async, but it is also much simpler to code than
> fully-async.

The problem with semi-async is that the developer pretends that his code
is linear, and he ends up expecting that. But it isn't, in general he is
just fooling himself in thinking he does not need to protect memory or
other resources.

The ugly code to handle the uid switching problem in nested calls is but
just one example in what goes wrong with semi-async. I tend to believe
that if the code was fully async that problem would have come up a lot
earlier, as soon as you had to return from a function and wait for a
callback, as then you'd have seen that you are returning control to the
main loop and the uid is not the right one.

As I see it, this is a problem of what you expect vs what you get.
And if what a programmer expect and what he gets does not align much
then you get into the madness I am talking about.

> If we are going to set the standard as "full-async or bust" then the
> project is doomed. The whole code base will end up like the ldb code,
> with most of the code handling the manipulation of callbacks.

What we need is better API that minimize the impact of full async, so
that you don't need to focus as much on the async aspect. Once you get
in the habit of dealing with it, it becomes much simpler.

> Is it hard to debug? sure it is. So is fully-async code. So is every
> concurrency system I've ever worked with. Concurrency is damn hard to
> do right.

Yes, and you better make it very explicit to the programmer that he is
dealing with concurrency, not give him a false idea of linearity. (A
very big IMO here).

> There are good reasons for not using the semi-async approach in many
> cases, but this treatment of it as radioactive is just silly.

The problem as I see it is that every time I look at how we all used
semi-async calls in s4, I see so many things that can go wrong, and that
are not accounted for in the code (and it would be harder to account for
than making the code fully async).
In another project, where using synchronous ldb calls was admittedly
easier, I simply took the route of creating a private event context for
that database, so that while ldb_wait() loops nothing else can happen.

>  > I fully agree, if you decide to go async, than so be it, make the code
>  > fully async.
> 
> for library calls, sure. For all the mainline code like the VFS? We
> should be fully async for the main async logic (such as blocking file
> io, notify events, oplocks etc), but to make every possible blocking
> point fully-async will just make the code inpenetrable.

As I see it, semi-async makes the code extremely fragile.

I do see your point, and I agree that fully async code is harder, and
maybe we just painted ourselves between a rock and hard place.

For my part, I will keep trying to make stuff as async as possible.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>