ccan dependencies seems to break external projects

Tue Feb 21 14:30:39 MST 2012

On Tue, 2012-02-21 at 08:36 -0500, Stephen Gallagher wrote:
> On Sun, 2012-02-19 at 10:01 +1100, Andrew Bartlett wrote:
> > On Sat, 2012-02-18 at 16:40 -0500, simo wrote:
> > > On Sat, 2012-02-18 at 11:08 +1100, Andrew Bartlett wrote: 
> > > > On Mon, 2012-02-13 at 08:04 -0500, Stephen Gallagher wrote:
> > > > > On Mon, 2012-02-13 at 08:08 +1100, Andrew Bartlett wrote:
> > > > > > On Sun, 2012-02-12 at 19:08 +0100, Jelmer Vernooij wrote:
> > > > > > > On 02/12/2012 06:57 PM, Matthieu Patou wrote:
> > > > > > > > On 02/12/2012 04:52 AM, Jelmer Vernooij wrote:
> > > > > > > >> On Sat, Feb 11, 2012 at 02:49:20PM -0800, Matthieu Patou wrote:
> > > > > > > >>> I'm trying to compile openchange as described here : 
> > > > > > > >>> http://tracker.openchange.org/projects/openchange/wiki/HowTo_Install_OpenChange_From_Source
> > > > > > > >>>
> > > > > > > >>> I'm using the latest git tree instead of alpha17, after a couple of
> > > > > > > >>> tweak in the script that setup the prerequisits I have a samba4 with
> > > > > > > >>> using system talloc/tdb/tevent and ldb.
> > > > > > > >>>
> > > > > > > >>> Then I try to build openchange and got:
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> /usr/local/src/openchange$ make
> > > > > > > >>> Linking sample application bin/libmapixx-test
> > > > > > > >>> g++ -I.  -O3 -Wall -g3 -fstrict-aliasing -Wp,-D_FORTIFY_SOURCE=2
> > > > > > > >>> -DHAVE_IMMEDIATE_STRUCTURES=1 -D_GNU_SOURCE=1
> > > > > > > >>> -Wl,-rpath,/usr/local/samba/lib -I/usr/local/samba/include
> > > > > > > >>>     -I/usr/local/samba/include   -I/usr/local/samba/include
> > > > > > > >>> -I/usr/local/samba/include     -o bin/libmapixx-test
> > > > > > > >>> libmapi++/tests/test.cpp libmapipp.so.0.11 libmapi.so.0
> > > > > > > >>> .11 -Wl,-rpath,/usr/local/samba/lib -L/usr/local/samba/lib -ldcerpc
> > > > > > > >>> -ldcerpc-binding -lndr -lsamba-hostconfig -lsamba-util -ltevent
> > > > > > > >>> -ltalloc   -Wl,-rpath,/usr/local/samb
> > > > > > > >>> a/lib -L/usr/local/samba/lib -lldb -ltalloc
> > > > > > > >>> -Wl,-rpath,/usr/local/samba/lib -L/usr/local/samba/lib -ltalloc
> > > > > > > >>> -lpthread
> > > > > > > >>> /usr/local/samba/lib/libsmbconf.so.0: undefined reference to
> > > > > > > >>> `hash_any at SAMBA_4.0.0ALPHA18_GIT_0B3A262'
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> The hash_any symbol comes from the ccan library and is included in
> > > > > > > >>> libsmbconf.so via source3/util/
> > > > > > > >>>
> > > > > > > >>> I suspect there is a symbols collision when you build samba 4 with
> > > > > > > >>> system libs as ldb has its own version of ccan but obviously with a
> > > > > > > >>> different version although it seems that ldb didn't directly need
> > > > > > > >>> ccan (no ccan in the output of ldd libldb.so), the following command
> > > > > > > >>> ls /usr/local/systemsamba/lib/ldb/libccan.so.0* output this:
> > > > > > > >>>
> > > > > > > >>> /usr/local/systemsamba/lib/ldb/libccan.so.0
> > > > > > > >>> /usr/local/systemsamba/lib/ldb/libccan.so.0.1-init-1161-g661d41f
> > > > > > > >>>
> > > > > > > >>> There is no way to have a system ccan and it seems that there is no
> > > > > > > >>> easy way to use the one built for ldb as the "system" ccan.
> > > > > > > >>>
> > > > > > > >>> I don't know what can be done, the cheapest is to have a
> > > > > > > >>> source3/util not to depend on ccan if --disable-tdb2 was specified
> > > > > > > >>> on the command line.
> > > > > > > >> Does ccan really have to be a separate library? Its goal seems to be
> > > > > > > >> the same as the 'samba-util' library.
> > > > > > > >>
> > > > > > > >> If I understand correctly ccan isn't really meant as a shared library
> > > > > > > >> with utility functions but rather as a collection of C files that you
> > > > > > > >> can pick a few useful ones from.
> > > > > > > >>
> > > > > > > >> Would it perhaps be possible to just include the relevant files from
> > > > > > > >> ccan in libsamba-util?
> > > > > > > > I'm not sure how this will fix the problem.
> > > > > > > > Let say that we have ccan in samba-util, instead of depending on ccan, 
> > > > > > > > ldb will depend on samba-util.
> > > > > > > > It means if I understand well that we will have a libsambautil.so with 
> > > > > > > > a version 1.4.1 (the version of ldb) but as it seems that we have no 
> > > > > > > > way to use a system samba-util library when building samba4 will have 
> > > > > > > > also a version of this library SAMBA_4.0.0ALPHA...
> > > > > > > tdb can include whatever it needs from ccan in the same way as 
> > > > > > > samba-util does (AFAIK it only uses a single macro from ccan at the moment).
> > > > > > > 
> > > > > > > ccan doesn't have a well-defined ABI. The problem with the way you're 
> > > > > > > building ldb and Samba is that you end up with two versions of the ccan 
> > > > > > > library - one which does have hash_any and one which doesn't. The linker 
> > > > > > > only tries to load one and not the other, because it remembers it has 
> > > > > > > already loaded ccan.
> > > > > > 
> > > > > > Does removing the 'vnum' line from ccan/wscript fix any of this?
> > > > > > 
> > > > > > It seems odd to be forcing the vnum from the upstream ccan if this
> > > > > > really is a private library?
> > > > > > 
> > > > > > Also, shouldn't we be putting a -samba4 prefix on the end of the name of
> > > > > > all the private and bundled libraries?  (We do it when bundling
> > > > > > Heimdal).
> > > > > > 
> > > > > > Andrew Bartlett
> > > > > > 
> > > > > 
> > > > > I don't think that encoding the version number in the library is a
> > > > > sufficient answer since, as discussed earlier in this thread, it is
> > > > > possible to build subsets of CCAN. This would presumably be the same
> > > > > library version but would be notably missing some components and
> > > > > therefore ABI-incompatible with an application linked against it.
> > > > > 
> > > > > There really only seem like two options:
> > > > > 1) CCAN needs to be modified so that we always build ALL components of
> > > > > it into a shared library. Ideally we would do this upstream with CCAN
> > > > > (and not privately in samba). Then we could have upstream release
> > > > > tarballs with a defined version that samba should then link against. I'm
> > > > > aware that Samba likes to have the ability to also build it in-tree, but
> > > > > that's a secondary concern. This option allows distributions to package
> > > > > a system library version of ccan (which is the preferred approach for
> > > > > most distributions, since it allows easy upgrades to fix bugs and
> > > > > security issues across all platforms).
> > > > > 
> > > > > 2) If CCAN truly cannot be built as a common library (like libreplace
> > > > > cannot), then the only correct answer should be to disallow ever
> > > > > building it as a shared object. We should allow only static linking (or
> > > > > pure copylib, like libreplace). If we're going purely copylib, we should
> > > > > remove the build-system entirely so that it can only be used by
> > > > > incorporating it into the Makefile/WAF build of its consumers. In other
> > > > > words, we should make it impossible to produce a system library or
> > > > > shared object that could ever conflict with the internal version.
> > > > > 
> > > > > I'm personally in favor of option 1 if it is at all possible.
> > > > 
> > > > 3) Install bundled libs in the private dir (as we currently do) with a
> > > > -samba4 suffix like we do with Heimdal.  I thought we used to do this
> > > > with all our bundled libs, but there has been a number of changes and
> > > > different approaches here over time.  This will also fix the popt
> > > > issue. 
> > > 
> > > This is not an acceptable option for many distributions. You need to
> > > jump thorugh hoops to justify why you want to use a private copy of a
> > > library instead of the version in the distribution, and for good
> > > reasons.
> > 
> > Well we can't do 2) either, as then we will end up with duplicate
> > symbols, which our build system correctly denies.  It would need to
> > remain a private Samba library, like all the others.
> > 
> > Rusty will have to indicate if he wishes to make ccan upstream a .so
> > with an ABI.  In the absence of that, I cannot see any alternative to
> > 3).  This at least also helps other non-linux, non distributions systems
> > that may get popt multiple times as well. 
> > 
> > In the meantime, for the release I hope to make very shortly,
> > --disable-tdb2 seems the only option for distributions with these
> > (reasonable, just difficult for rusty's ccan modal) rules. 
> > 
> > Andrew Bartlett
> > 
> 
> Adding Rusty explicitly to get his opinion.
> 
> Obviously, I'm in favor of option 1) as I think it's the *correct*
> answer for packaging on most distributions. It's unhealthy to have
> bundled libs lying around, since it means you need to coordinate
> multiple package updates in the event that a major bug or security issue
> is discovered.

I hate the idea of pushing this on Rusty (as 'only' a user of ccan), but
unless we do 1) we need to address the reason we removed -samba4 (etc)
suffixes per:

http://lists.samba.org/archive/samba-technical/2011-January/075816.html

That is, for 3) we would need to declare that popt and ccan both do not
include static pointers, and are therefore safe to include multiple
times in the same codebase.  But I've seen Rusty burn right though his
allowance of 'special changes for for samba' patience on all our other
requirements, so I'm reluctant to ask even more of him. 

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org