samba4 irc discussion

[Stefan (metze) Metzmacher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi *,

here a log from a discusion with tridge on IRC:
(and I still think IRC is very usefull, otherwise this has taken month to discuss that
~ via email:-)

metze

[07:34:10] tridge [~tridge at 203-173-8-192.perm.iinet.net.au] has joined #samba-technical
[07:34:10] ChanServ sets mode  #Samba-technical +o tridge
[07:37:05] <metze> hi tridge
[07:37:38] wasabi [~wasabi at c-67-162-246-69.client.comcast.net] has joined #samba-technical
[07:37:39] <@tridge> hi
[07:37:52] <wasabi> low
[07:37:56] <@tridge> metze: using --enable-debug solves the problem - the -O2 goes away
[07:38:11] <@tridge> what I don't understand is why -O2 was appearing in the first place
[07:40:32] <wasabi> one of you might know this... i'm having trouble finding out. is seteuid per
"process" or per pid (thread? etc? whatever?)
[07:42:29] <@tridge> depends on the Linux flavour
[07:42:35] <wasabi> super. ;)
[07:42:43] <@tridge> in linux it is per-thread I believe (at least last time I tested)
[07:43:05] <wasabi> that's what I hoped. I'm working on a methodology for "impersonation" under unix.
[07:43:09] <@tridge> i'm not sure if the latest pthreads specs say which is correct
[07:43:31] <@tridge> but I'd strongly suggest you test this for yourself
[07:43:42] <@tridge> its quite possible that NPTL until linux changes the behaviour
[07:45:26] <metze> tridge : how can I reproduce that I get -02 there
[07:46:03] <@tridge> just use ./configure --enable-developer
[07:46:15] <@tridge> and --enable-experimental
[07:46:28] <@tridge> not sure if --enable-experimental is needed ... i'll check
[07:47:20] <mimir> hi metze, tridge
[07:47:28] billj [~newbie at 203.101.103.131] has quit irc ("KVIrc 3.0.0-beta2 "T-Rex"")
[07:49:13] <@tridge> metze: ok, "./configure --enable-developer" gives -O2
[07:49:22] <@tridge> the question is, where is the -O2 coming from?
[07:50:33] <metze> will you be on irc a bit longer today?
[07:50:49] <@tridge> yep
[07:51:09] <metze> I just run svn up and will check
[07:52:36] <metze> what does 'echo $CFLAGS' gives on your machine
[07:53:40] <@tridge> nothing
[08:00:04] <metze> what do you aspect with just ./configure?
[08:00:29] <metze> don'T try, just say what you think
[08:07:55] <@tridge> with ./configure I would expect -O2
[08:08:02] <metze> why?
[08:08:05] <@tridge> the problem isn't really to get rid of -O2
[08:08:17] <@tridge> the problem is to understand how the new build system decides these things
[08:08:41] <@tridge> i have solved the immediately problem with --enable-debug, but I am a bit
frustrated that I don't understand how it works
[08:09:36] <metze> maybe it's a bug I introduced when changed to don'T use the developer CFGLAGS
while we run the configure tests
[08:10:02] <@tridge> could be
[08:10:23] <metze> where is this -02 comming from?
[08:10:24] <@tridge> but more generally I'd like to understand "how does the build system construct
the CFLAGS to use?"
[08:10:42] <@tridge> have you reproduced it btw?
[08:10:47] <metze> tridge :  yep
[08:10:58] <@tridge> ok, so its not my imagination then :)
[08:11:06] <metze> :-)
[08:12:08] <metze> I have -g -O2 with ./configure
[08:12:15] <@tridge> right
[08:12:34] billj [~newbie at 203.101.103.131] has joined #samba-technical
[08:19:22] <metze> tridge: the CFLAGS are created the same way as in samba3 currently
[08:20:00] <@tridge> no they aren't - it isn't done in configure.in :)
[08:20:34] <metze> but every test just add the flags to the GLOBAL CFLAGS var
[08:20:56] <metze> and ACSUBST(CFLAGS) replace @CFLAGS@ in Makefile.in
[08:21:00] <@tridge> yes, but I haven't found the line in any source file in samba4 that actually
adds it
[08:21:11] <@tridge> I spent ages doing grep -r on various things, and can't find it
[08:21:31] <metze> what adds what?
[08:21:36] <@tridge> so, the Q is, where is the string "-O2" coming from that is ending up in the
Makefile
[08:22:23] <@tridge> (and btw, in current samba3 we don't end up with -O2 with --enable-developer)
[08:22:56] <metze> but without
[08:23:09] <metze> and where is this comming from?
[08:23:13] <@tridge> let me try to explain again from the beginning
[08:23:25] <@tridge> the generated Makefile contains the string -O2
[08:23:35] <@tridge> that string must come from _somewhere_ in the source tree
[08:23:50] <@tridge> the task is to find what file/line contains the string "-O2"
[08:24:11] <metze> please first explain me where this is comming from in samba3
[08:24:22] <@tridge> ill go look
[08:24:32] <metze> I found only one line
[08:25:01] <metze> line 1203 in configure.in
[08:25:27] <metze> that is an aix case which sets PICFLAGS="-O2"
[08:25:45] <@tridge> hmm, you're right, so is it autoconf doing this?
[08:26:13] <@tridge> is it some system macro?
[08:26:25] <metze> I don't know, maybe one of the first tests
[08:26:44] <metze> when CFLAGS is still empty
[08:26:49] <@tridge> hang on - somehow --enable-developer disables it in samba3
[08:26:56] <metze> I know
[08:27:04] <@tridge> ok, i understand now
[08:27:13] <@tridge> --enable-developer overrides CFLAGS
[08:27:18] <@tridge> removing -O2
[08:27:24] <metze> no
[08:27:26] <@tridge> the Q is, why doesn't this happen in samba4
[08:27:38] <metze> I know this
[08:27:40] <@tridge> autoconf is supplying a default CFLAGS
[08:27:43] <metze> let me explain
[08:28:17] <metze> so in samba3 we set the CFLAGS with the developer flags on top of configure.in
[08:29:04] <metze> when we then reach what every the is in autoconf that add the default CFLAGS if
there empty
[08:29:17] <metze> it will don'T change the cflags
[08:29:28] <metze> because there are already set
[08:30:03] <metze> and in samba4 we add the developer flags add the end of configure
[08:30:35] <metze> because I added -Werror-implicit-prototype
[08:30:56] <metze> and that caused some configure tests to fail
[08:31:25] <metze> so I decided to first check if the compiler understands this flags
[08:31:43] <metze> and then later add them when the configure script is at the end
[08:31:56] <@tridge> the compiler understands it, but you can't use that flag duing configure
[08:32:03] <metze> yep
[08:32:05] <@tridge> as the tests are not written to ensure full prototypes
[08:32:19] <metze> yep
[08:32:42] <metze> maybe I'll revert this change and take another approach
[08:33:18] <metze> but with this it was possible to use --enable-developer on compilers that are not gcc
[08:33:28] <@tridge> teaching programmers to read warnings would be a good start :)
[08:33:36] <@tridge> lots of recent patches have ignored warnings
[08:33:48] <metze> yep
[08:34:35] <metze> tridge : I really think that all the mess you had with the build system is
because it's not completely finished,
[08:35:02] <@tridge> oh, quite possibly, and I think that in many areas its a big improvement
[08:35:16] <metze> and the stuff that confuses everybody is the rest of the samba3 mess
[08:35:23] <@tridge> i just worry about the "what if metze is run over by a bus" problem
[08:35:49] <metze> that's there because I can't do all at once
[08:35:51] <@idra> metze, can you just add some explanatory test to the build directory?
[08:36:18] <@idra> metze, I looked at the code and understood "something" but it is way complicated
[08:36:52] <metze> idra: I'll do
[08:37:06] <@tridge> meanwhile, look carefully before crossing the road :-)
[08:37:13] <metze> :-)
[08:38:03] <metze> maybe a README that says look at the top of build/smb_build/public.m4
[08:38:10] <metze> would be a good idea
[08:38:41] <@tridge> one thing that would be nice is a auto-generated comment at the top of the
generated files saying "this was generated from file FOO.BLAH"
[08:38:45] <metze> because that's what the most people need to know when they want to use the build
system
[08:39:31] <metze> tridge : what files to you have in mind?
[08:39:45] tarjei [~tarjei at mail2.bergfald.no] has joined #samba-technical
[08:39:49] <metze> when you look through config.smb_build.pl
[08:40:06] <metze> you'll see #line 1 "build/smb_build/config_mk.pl"
[08:40:09] <metze> lines
[08:40:35] <@tridge> hmm, I'd missed those
[08:40:54] <metze> and when there's an perl error it displays the correct line in the source file
[08:41:04] <@tridge> ok
[08:41:19] <@tridge> maybe I just need to spend more time with it
[08:41:31] <metze> maybe I can find more places to add something like this
[08:42:03] <metze> but that goes all back to the problem where is what coming from and why
[08:42:25] <metze> and this can't be solved till the build system is complete
[08:43:36] <@tridge> ok, i'm happy to go back to lurk-and-wait mode :)
[08:43:44] <metze> :-)
[08:44:27] <metze> tridge : I only want first to get the source code infrastructure right
[08:45:06] <metze> (generic sockets, datagram services..., client lib code, ...)
[08:45:35] <@tridge> yep - and good progress is being made on that
[08:45:55] <metze> so that other people can start really working on backends, without having
infrastructure changes every few days
[08:46:23] <metze> when that's done, I'll return to the build system
[08:46:57] <metze> my goal is that we don't need to run make clean anymore
[08:47:20] <metze> just make should rebuild all files which are need for a consistent build
[08:50:08] <metze> tridge : what are your next plans?
[08:51:21] <@tridge> catching up on some emails, finally doing the async rpc libs for andrew,
starting on the posix NTVFS and getting the IDL license sorted out
[08:51:43] <@tridge> i only arrived back home last night
[08:55:17] <metze> about the client libs, I would like applications to use there own event loop
[08:55:57] <metze> and maybe there own memory management, and there own debug functions
[08:57:08] <metze> I thought about a GDEBUG(ctx, level, ("text"));
[08:57:55] <@tridge> what did you think of the inherited event loop context I used in libcli? Is
that the approach you are thinking of?
[08:58:30] cat__ [~cat at 217.21.35.33] has quit irc (Remote closed the connection)
[08:58:57] <metze> tridge : yep, but the client app should be able to select a custom
event_loop_wait() implementation
[08:59:40] <metze> I thought having a event_set_loop_wait_fn() would be good
[09:00:01] <metze> and the current one is the default
[09:00:01] <@tridge> do you have an application in mind for this?
[09:00:28] <metze> no, just any application which want to use our shared libs
[09:01:21] <metze> I think when we provide shared libs, we need to think carefull about this stuff
before we release them the first time
[09:01:25] <@tridge> then you'd also need the application to be able to override the code to setup
event contexts
[09:02:01] <metze> maybe, but I think that's not completey needed
[09:02:28] <metze> they can write a wrapper around the samba event structs
[09:03:19] <metze> ...we should try to break compat after releasing shared libs
[09:03:32] <metze> ...we should try to *not* break compat after releasing shared libs
[09:03:36] <metze> upps:-)
[09:03:37] <@tridge> :)
[09:04:57] <metze> tridge: when I do a talloc(NULL,50); is there a DLIST_* operation called?
[09:06:37] <@tridge> its like a malloc() in that case
[09:06:46] <@tridge> currently no globals in talloc
[09:07:20] <@tridge> tridge at blu:~/samba/samba4/source$ size lib/talloc.o
[09:07:20] <@tridge>    text    data     bss     dec     hex filename
[09:07:20] <@tridge>    3702       0       0    3702     e76 lib/talloc.o
[09:07:47] <metze> in samba3 ther were globals, right?
[09:07:52] <@tridge> yes
[09:07:57] <metze> ok
[09:08:10] <@tridge> and if we want to support the "signal causes dump of memory usage" then we will
need globals again
[09:08:19] <@tridge> i removed that in the rewrite
[09:08:53] <metze> hmm, not sure if that's a good idea
[09:09:04] <metze> to readd globals
[09:11:01] <metze> what's about refcounts in talloc, would it maybe make it easier to use refcounts
there instead of using them in the upper layer structs?
[09:11:01] <@tridge> the signal is v useful
[09:11:16] <@tridge> the other alternative is to use the tree nature of talloc
[09:11:26] <@tridge> and create a "per-process" top level talloc ctx
[09:11:36] <@tridge> then make talloc(NULL, x) derive from that
[09:11:43] <@tridge> the signal would then operate on that ctx
[09:12:19] <metze> and what if we have multiple thread in a process?
[09:12:34] <@tridge> then we'd need to make it per-thread
[09:12:41] <@tridge> ie. it would be tied to the process model
[09:15:45] <metze> ok
[09:16:21] <metze> but how talloc(NULL,5) find the correct parent?
[09:16:32] cat__ [~cat at 217.21.35.33] has joined #samba-technical
[09:16:55] <@tridge> we'll need a process-model aware way of finding a global
[09:17:31] <metze> ok
[09:17:36] <@tridge> we could possibly use __thread for threads
[09:17:49] <@tridge> if the compiler supports it
[09:18:13] <metze> btw: I'm currently doing union smb_negprot
[09:18:51] <@tridge> ok, thanks!
[09:19:10] <@tridge> i was lazy with that one
[09:20:05] <metze> :-)
[09:21:02] <@tridge> another problem with libcli is that a couple of routines still cannot be used
async
[09:21:21] <@tridge> in particular, negprot, session setup and the directory listing routines
[09:21:30] <metze> maybe having print functions for the smb stuff would be also usefull, like
NDR_PRINT_DEBUG()
[09:21:35] <@tridge> we should have _send versions of these
[09:21:41] <@tridge> yes
[09:21:56] <@tridge> its possible we could auto-generate those
[09:22:17] <@tridge> i think that it would be possible to write a parser that could parse
smb_interfaces.h
[09:22:31] <metze> yep
[09:22:58] <metze> you were faster in typing that, I had the same idea
[09:23:15] <@tridge> doing a code generator would be harder
[09:23:21] <@tridge> but a print fn is quite easy
[09:23:37] <metze> what I also had in mind was a new version of mkproto.pl
[09:23:44] <@tridge> oh?
[09:24:12] <metze> which also greps the comments above a prototype and include that into the
generated header
[09:24:28] <@tridge> hmm, maybe
[09:26:06] <metze> ...but you don't want to hear about my plans with seperate proto headers and
private and public onces, right?
[09:26:29] <@tridge> i'll listen, but i won't be easy to convince :)
[09:26:38] <metze> :-)
[09:27:06] <metze> for a first step I just want to generate them seperately
[09:27:41] <metze> but also generate a file that includes all of them, so that logicaly end with the
same we have now
[09:28:08] <@tridge> what does that gain us?
[09:30:35] <metze> this 1 step nothing:-)
[09:30:43] simo [~simo at nat.xsec.it] has joined #samba-technical
[09:30:48] <metze> but the steps that will follow
[09:31:35] <metze> e.g. that we only need to rebuild a proto header when a source file was changed
[09:32:18] <metze> so no explicit make proto or make clean, should be needed
[09:32:30] <metze> that will also fix the make -j 4 problem
[09:32:33] <@tridge> that is not necessarily a good thing
[09:32:41] <metze> why?
[09:32:47] <@tridge> -j fix is good, but no "make clean" can be bad
[09:32:55] <metze> why?
[09:33:13] <@tridge> because if you take it to its logical conclusion, then just about any C file
change requires a "make clean"
[09:33:24] <@tridge> and that gets very tedious
[09:33:38] <@tridge> ok, say you change a C file foo/bar.c
[09:33:54] <@tridge> and you recompile just that C file - then what if some other C file called the
function you changed?
[09:34:27] <@tridge> then that C file needs recompiling too, as a change in function definition can
cause a change in caller behaviour in C
[09:34:44] <@tridge> due to auto-casting
[09:34:53] <@tridge> so basically its impossible to do this right in C
[09:35:17] <metze> what do you mean by function definition
[09:35:35] <@tridge> ok, say you change a function from taking "int" to taking "long"
[09:35:43] <@tridge> or from size_t to off_t
[09:35:46] <metze> then the prototype changes
[09:35:52] <@tridge> then all callers need to be recompiled
[09:36:04] <@tridge> right, so anyone including that proto header needs recompiling
[09:36:16] <metze> that would cuase a new make proto and the other file depending on this proto file
is reabuilded
[09:36:24] <metze> yep
[09:36:31] <metze> that's the idea
[09:36:40] <@tridge> so, if the proto file is included globally, then thats a full recompile
[09:37:11] <metze> but if we have seperate proto files not
[09:37:19] <@tridge> and then we lose PCH
[09:37:41] <@tridge> plus, we get the huge include mess that so many projects get into
[09:37:42] <metze> and the global proto file which include the others is not changed
[09:38:06] <metze> tridge: we just still include one file per c file
[09:38:25] <metze> so globa_proto.h includes all others
[09:38:26] <@tridge> in that case, how do you know who is calling the function?
[09:38:33] <@tridge> you have to assume everyone is
[09:38:50] <@tridge> as everyone has the prototype
[09:39:15] <metze> we still didn'T reach the final state
[09:39:25] <simo> tridge, I'm going to test a new case for strndup, do I still have access to you
AIX box ?
[09:39:35] <@tridge> simo: yes
[09:39:40] <simo> thanks
[09:39:40] <metze> hi simo
[09:39:49] <@tridge> simo: bug has been submitted to two different groups on AIX libc team
[09:40:18] <@tridge> metze: the basic problem with auto-rebuild is that it has to be conservative
[09:40:32] <metze> tridge : I know
[09:40:40] <@tridge> and as a day-to-day developer I find it very annoying when a large rebuild
happens for a small change
[09:41:20] <metze> tridge : I have thought of all that
[09:41:40] <@tridge> then convince me :-)
[09:41:53] <metze> :-)
[09:42:04] <metze> step-by-step:-)
[09:43:00] kinkie [~kinkie at 194.244.5.6] has joined #samba-technical
[09:43:10] <metze> note that I'm sitting in the same boat as you, and don't want to wait when I
compile samba:-)
[09:43:43] <metze> so it costs me a lot of time when I need to do a make clean;make
[09:43:47] <@tridge> yes, we just don't agree on the right life-jacket to use :)
[09:43:56] <metze> :-)
[09:44:47] <metze> tridge : I think it would be good to mail a log of this long discussion to
samba-technical to make jra a bit more happy:-)
[09:45:08] <@tridge> sure

- --
metze

Stefan Metzmacher <metze at samba.org> www.samba.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3-nr1 (Windows XP)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBKvKzm70gjA5TCD8RApN5AKCMwGIB/JvCsl/2rKAl2Yf4bRraVACfe/kV
0b7qTJkwXGD8wdlUl6WY3Sk=
=v2gA
-----END PGP SIGNATURE-----