tdb API issues
Howard Chu
hyc at highlandsun.com
Thu Apr 2 13:25:14 GMT 2009
tridge at samba.org wrote:
> Hi Howard,
>
> > I'd thought of that at first, but it seems that a separate context per thread
> > would also multiply the address space consumed per DB, since they would each
> > have their own complete mmap of the data. Doesn't sound too practical.
>
> It would multiply the virtual address space used, but not the physical
> memory used. Do you think thats really a problem? How many threads do
> you have?
By default we use 16 threads. More on larger SMP systems. Currently in a 32
bit OS there's a practical limit of about 1 million entries for a completely
in-memory DB. (1M ~ 20 bits, average entry size ~1K : 10 bits, 30 bits of
address space used right off the top, +/- other overheads...) If we have a
separate map per thread then our max DB size is drastically reduced, which
drastically reduces the range of applicability of this solution. We don't need
this to scale to billions and billions of entries, but it'd be nice to cover
the hundreds of thousands; that would make it viable for a large portion of
our user base.
> It might also be possible to have a common virtual address space. To
> do that we'd break up the tdb_context structure into per-thread and
> per-process parts, and put the mapped pointers in the per-process
> part. It would require some thought to make sure this is safe, but at
> first glance I think its doable.
OK, this sounds like a reasonable avenue to explore. If we also provide some
callbacks for creating, locking/unlocking and freeing mutexes then we can
explicitly make the relevant parts safe.
simo at samba.org wrote:
> You should be able to change the tdb_context init code so that all
> contexts will actually use the same mmap space.
> You must open the file just once anyway otherwise you loose all locks if
> any thread closes the file.
Right.
> To be honest, I think Tridge is right in proposing to have a context per
> thread. It's probably a very good way to achieve what's needed without
> having to change the API (may require linking in pthreads), or with
> minimal additions (additional API used just by threaded processes to set
> up per thread access).
Yes, as long as we keep everything using the same mmap it would be fine.
diego.zuccato at unibo.it wrote:
> What about using a single thread for tdb access?
Given the rise of multicore processors, I think this is out of the question
for us.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
More information about the samba-technical
mailing list