talloc: talloc_set_memlimit causes all reallocs to fail when used on pools. talloc_set_memlimit not enforced correctly on pools.

[Jeremy Allison]

On Mon, Oct 19, 2020 at 11:17:29PM -0500, Arran Cudbard-Bell via samba-technical wrote:
> Much appreciated!  I'll try this out tomorrow.
> 
> Just so I'm clear on the expected behaviour, could
> you verify that these assumptions are correct:
> 
> 1. If there's a 1024 byte pool, with a 2048 byte memlimit,
> the amount of memory allocated outside of the pool would 
> be limited to 1024 bytes.

Yes. Pool == 1024 bytes allocated, any extra must be
below total size (including pool, which is treated
as allocated).

> 2. If there's a 1024 byte pool, with a 1024 byte memlimit,
> any alloc or realloc up to the pool size would succeed,
> but any alloc or realloc which'd result in a chunk being
> allocated outside of pool memory would fail.

Yes. memlimit == pool size, so any further mallocs/reallocs
adding memory should fail.

> 3. If there's a 1024 byte pool, with a 512 byte memlimit,
> the memlimit would cause all reallocs and allocs to
> fail, because the pool size is already greater than the
> memlimit.

No. Allocations inside the pool below 1024 bytes would
always succeed (until the pool is filled) as when allocating
from a pool the memlimit isn't checked if the allocation
can be filled from within the pool.

memlimit is only checked in actual malloc/realloc calls.

If a pool already exists, it's assumed that it must already
have fit within the memlimit. It makes no sense to
set the limit lower than the pool size.

> Additionally, if the third assumption is correct, do you 
> see any value in instead having talloc_set_memlimit 
> fail if a caller specifies a memlimit smaller than the pool?

Nope. See above.