[PROPOSAL] Re-bundle (stop producing tarballs for) ldb?

[Simo]

On Thu, 2019-04-11 at 07:36 +1200, Andrew Bartlett via samba-technical
wrote:
> On Mon, 2019-04-08 at 13:11 +0200, Andreas Schneider wrote:
> > On Monday, April 8, 2019 11:15:58 AM CEST Alexander Bokovoy via
> > samba-
> > technical wrote:
> > > On ma, 08 huhti 2019, Andrew Bartlett via samba-technical wrote:
> > > > ldb 2.0.0 was marked in master just today, not out of some great
> > > > fanfare but because an externally exposed API/ABI changed and the
> > > > .so
> > > > naming rules require it (as we strictly bind the two things for
> > > > simplicity).
> > > > 
> > > > ldb has been a great project, but aside from sssd and the now
> > > > defunct
> > > > openchange, it simply hasn't taken off, certainly not in the way
> > > > that
> > > > talloc and tdb have become central parts of the Linux ecosystem.
> > > > 
> > > > Samba development and Samba AD DC needs drive ldb, and it isn't
> > > > going
> > > > to be an independent project any time soon.
> > > > 
> > > > So, I'm wondering if we should stop producing ldb tarballs.  We
> > > > already
> > > > have to bump version numbers strictly branch with Samba releases
> > > > and
> > > > have complex build logic to ensure we don't build with the wrong
> > > > version.
> > > > 
> > > > For Samba the ABI has always turned our to be a tricky beast,
> > > > even
> > > > outside the module stack (where no promies were made) we quickly
> > > > found
> > > > that using Samba with the 'wrong' ldb version was just looking
> > > > for
> > > > pain.  
> > > > 
> > > > After a muck-up where a master version of ldb was published into
> > > > debian
> > > > (and so breaking existing setups), our distributors have wondered
> > > > the
> > > > same as well.  Specifically I recall a discussion on the debian
> > > > packaging list about if the ldb package should just be built from
> > > > the
> > > > Samba tarball instead.
> > > > 
> > > > So this is my proposal: that we build ldb like we build libndr
> > > > and
> > > > libsmbclient.  Others can still build against it as a public
> > > > library,
> > > > but we never build against a 'system' version.  We should have an
> > > > option to keep it private as well, just like the current default
> > > > build.
> > > > 
> > > > What do others think?
> > > Looks like that in the past a need for many breaking changes in
> > > libldb
> > > was driven by an incomplete design of the features for Samba AD.
> 
> I'm sorry, can you be more specific?
> 
> > > Perhaps, spending a bit more effort at a drawing board would have
> > > helped
> > > here rather than shipping and then rewriting the code. I'm trying
> > > to
> > > reflect on this, not blame anyone as one can always find a case
> > > where
> > > own hands were dirty as well.
> 
> Perhaps.  However the changes that have caused the most need to avoid
> version skew have not been those addressing new things, but fundamental
> mis-features from the very earliest days of ldb.  
> 
> I spoke on this at the linux.conf.au sysadmin miniconf in 2018:
> https://sysadmin.miniconf.org/2018/lca2018-andrew_bartlett-fixing_tridges_mistakes-taking_samba_ad_to_scale.pdf
> https://www.youtube.com/watch?v=JNR8bJkWCjA
> 
> I've lost the keys to the TARDIS, do you have them?

I do not think being sarcastic really helps anything Andrew.

> > Also there is a feature on most Unix systems called symbol
> > versioning. With 
> > this it is even possible to break the API if the library is
> > versioned 
> > correctly. See:
> > 
> > https://github.com/ansasaki/abimap
> 
> The biggest ABI-breaking changes that have been:
> 
> Samba 4.7 - per-search locking was missing/added at the ltdb layer
> Samba 4.9 - pre-search locking was missing/added at the ldb layer
> (protecting the module stack).
> 
> Samba master - change to a public structure
> 
> I'm not sure how any of these could have been handled, even at great
> engineering cost, via symbol versions.  

Well you need to start by considering the possibility when you engineer
a change. There are always changes that end up requiring a whole new
incompatible version, that happens, but they tend to be rare and game
changing.

> Beyond that, the fundamental challenge is that currently the ldb
> version is changed, and metze diligently tags a relaese for every minor
> ABI change.  
> 
> As I mention above:
>  - ldb for Samba is tested as a whole as part of Samba, and we rightly
> banned mixing ldb/samba versions due to the result being untested.
>  - ldb is made stable by the same process as the whole of Samba.  The
> intermediate releases, forced by an ABI change, are not helpful to
> users and are actively harmful for packaging teams, so we should not
> provide them.

I do not understand what this means.

> Finally, to address Simo's point about external contributions, I wish
> it were so.  Sadly for as far as I can determine contributions to ldb
> have been almost exclusively for:
>  - Samba's AD DC by members of the Samba team
>  - the unit tests by Jakub Hrozek
>  - python3
> 
> The contributor histogram for all time and the last four years is
> attached,  I may be missing something.

You are missing a lot, you may notice my (C) is sprinkled in quite some
places in ldb and yet I do not appear at all in your "history of all
times" ...

Number of patches mean little also, some people tend to splatter
patches all over with small changes in each and some people tend to
concentrate more fundamental changes in smaller patch sets.

You are also discounting many small but important contributions to the
stability of ldb brought in by downstream users (and often brokered by
me in earlier times) as unimportant. I find that dismissive and
arrogant, and completely inappropriate.

> I also don't see how the smaller tarball assists these in any case, the
> independent build system and ABI checks are still being maintained (as
> per ctdb) and all contributions have to go via CI and autobuild on the
> full git repo anyway.

The smaller tarball for once assures ldb can be built independently
without having to pull the whole of samba, that is very useful for
downstreams, just like for the tdb and talloc tarballs.

> Clearly I'm missing something:  Can you spell our more clearly how you
> see the production of the independent tarball at essentially random
> development intervals assisting ldb development?

It assist downstream deployment and consumption, which in turns feeds
back into development later by keeping us on our toes with ABI
stability and bug reports/patch feedback.

It's about maintaining an ecosystem.

Simo.