group mapping in Samba3

[simo]

On Tue, 2010-03-02 at 14:14 +1100, tridge at samba.org wrote:
> Hi Simo,
> 
> Sorry for not being able to finish our conversation on IRC earlier
> today. I had to run off to a meeting.
> 
> In regard to your patch be026a6fd89b44ba7e6bdf5bef049959b242c61e, I
> wonder if you have thought about the scaling and clustering issues?
> 
> Your commit comment indicates that you made this change as you wanted
> to make the default be cluster friendly. Unfortunately it also means
> that we're back to the original problem that led to the group mapping
> change 4 years ago, which is that the group mapping code now doesn't
> scale. It uses traverses to do lookups, because it has no indexing. I
> haven't run any profiling, so perhaps this is no longer an issue with
> the way idmap and related subsystems work in s3?

I haven't done extensive performance tests, but if we still have a
performance problem with tdb we need to fix it there so that the cluster
version benefits from it too.

> The next logical step beyond your current patch would be to
> re-introduce the multi-key tdb system that Volker worked on 4 years
> ago, but if you do, then please be aware that this is not quite as
> easy as it seems if your aim is for clustering. You could easily
> introduce deadlock scenarios because multi-key systems on top of tdb
> do not have predictable lock ordering if you use hash chain
> locks. Possibly clustered tdb transactions solve this?

I will work with CTDB people to find the right solution to this.

Keep in mind we are at the start of the journey for 3.6 (this was
committed only to master not any stable version) if it turns out that
the scalability problem is unbearable and for some reason we can't make
the (c)tdb code base address it properly we can always go back and
resurrect the ldb code.
Although I would probably rather have a conditional path within
mapping_tdb to build a custom index when ctdb is not used. Keeping all
the code in one place is just more maintainable than trying to keep 2
code paths that do basically the same thing.

Also because this was the only real s3 dependency on ldb, this means we
can more easily scrap the s3 copy of ldb and finally work to make the s4
ldb a common library for both. When that happen s3 will be able to use
the current ldb with all the fixes and improvements but avoid the nasty
double copy maintenance toll that didn't work.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>