Waf rebuilding issue with python3

[Stefan Metzmacher]

Am 12.12.18 um 17:55 schrieb Alexander Bokovoy via samba-technical:
> On ke, 12 joulu 2018, Volker Lendecke via samba-technical wrote:
>> On Wed, Dec 12, 2018 at 12:15:26PM +0200, Alexander Bokovoy via samba-technical wrote:
>>>> and that re-builds the world. So on my CPU it takes ages to get the
>>>> testenv up and running. And if I Control-C the "make testenv" it
>>>> always starts from scratch. So I have to really sit through the whole
>>>> build every time I want a testenv.
>>> I will be able to look at this on Thursday. 
>>
>> Is there anything you can recommend in the meantime short of working
>> on 4.9? I have some tricky code to push through autobuild (replacing
>> our DNS client), and having to wait 15 minutes for every "make
>> testenv" really slows me down. I hesitate to port my code to 4.9, but
>> if you say that this will persist I will do that and abandon master
>> for the time being.

Can you try commit 7d271450f71a168898482d8c8a0f1f6bbf875804?

> I tried building with your recipe (without spotlight support but that
> shouldn't matter) and what I found is that there is no change with
> subsequent make runs if I change a file like source3/smbd/smbd.h:

I also tried to reproduced it and it doesn't rebuild
for me.

It only rebuilds if I toggle between 'PYTHON=python2 make'
and 'PYTHON=python3 make'.

> $ diff -urN -q ~/tmp/samba-c4che{,-new}
> Files /home/abokovoy/tmp/samba-c4che/build.config.py and /home/abokovoy/tmp/samba-c4che-new/build.config.py differ
> Files /home/abokovoy/tmp/samba-c4che/_cache.py and /home/abokovoy/tmp/samba-c4che-new/_cache.py differ
> Files /home/abokovoy/tmp/samba-c4che/default_cache.py and /home/abokovoy/tmp/samba-c4che-new/default_cache.py differ
> Files /home/abokovoy/tmp/samba-c4che/sambadeps and /home/abokovoy/tmp/samba-c4che-new/sambadeps differ
> $ diff -urN -q ~/tmp/samba-c4che{-new,-2nd}
> 
> $ sha224sum ~/tmp/samba-c4che{,-new,-2nd}/sambadeps
> sha224sum: /home/abokovoy/tmp/samba-c4che/sambadeps: No such file or directory
> 2a6ad1603762d95b7d8696a8865166dab55efe5c6d978546d4bf5760  /home/abokovoy/tmp/samba-c4che-new/sambadeps
> 2a6ad1603762d95b7d8696a8865166dab55efe5c6d978546d4bf5760  /home/abokovoy/tmp/samba-c4che-2nd/sambadeps
> 
> ~/tmp/samba-c4che is bin/c4che content for ./configure.developer
> ~/tmp/samba-c4che-new is after first make run
> ~/tmp/samba-c4che-2nd is after I added a space to source3/smbd/smbd.h
> and re-run make
> 
> The second make run did re-run only parts including smbd.h and depending
> on those.
> 
> However, I tried few more cases like 'waf --show-deps smbd/smbd.objlist'
> and that one caused a full recompile. This time sambadeps differed:

What if you run 'python3 waf ...'?

> $ sha224sum ~/tmp/samba-c4che{-2nd,-3rd}/sambadeps
> 2a6ad1603762d95b7d8696a8865166dab55efe5c6d978546d4bf5760  /home/abokovoy/tmp/samba-c4che-2nd/sambadeps
> 287b0cf850f015d828a8bf4b6e74f11e89973365d4b40ac7f40707b2  /home/abokovoy/tmp/samba-c4che-3rd/sambadeps
> 
> save_samba_deps() is called from check_project_rules() if the target is
> not 'install', so we refresh them all the time. This was before as well,
> so perhaps a structure within the sambadeps is being inconsistent.
> 
> The only change we had recently for save_samba_deps() is by Metze but it
> shouldn't really affect generation of the deps structure.
> 
> I'm not sure why this happens, will be looking in more details tomorrow
> afternoon.
> 
> 
> commit 8e1135f99b34712fe4cce8039d2bb82df131ce40
> Author: Stefan Metzmacher <metze at samba.org>
> Date:   Thu Nov 15 11:41:07 2018 +0100
> 
>     wafsamba: remove the need of BuildContext.bdir
> 
>     Signed-off-by: Stefan Metzmacher <metze at samba.org>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> 
> diff --git a/buildtools/wafsamba/samba_deps.py b/buildtools/wafsamba/samba_deps.py
> index 542e682619c..d6b7c0f88d6 100644
> --- a/buildtools/wafsamba/samba_deps.py
> +++ b/buildtools/wafsamba/samba_deps.py
> @@ -1011,14 +1011,14 @@ def save_samba_deps(bld, tgt_list):
>          if tdeps != {}:
>              denv.outenv[t.sname] = tdeps
> 
> -    depsfile = os.path.join(bld.bdir, "sambadeps")
> +    depsfile = os.path.join(bld.cache_dir, "sambadeps")
>      denv.store_fast(depsfile)
> 
> 
> 
>  def load_samba_deps(bld, tgt_list):
>      '''load a previous set of build dependencies if possible'''
> -    depsfile = os.path.join(bld.bldnode.abspath(), "sambadeps")
> +    depsfile = os.path.join(bld.cache_dir, "sambadeps")
>      denv = ConfigSet.ConfigSet()
>      try:
>          debug('deps: checking saved dependencies')
> 

I don't think this is related.

Even commit 1dc06aaa2d6ceff494045034235aaf1c99252cf1
and git cherry-pick of 8ba0a9a1abb8bbe1df5ff808645adf305bc4e0b3
trigger a rebuild with ubutnu 18.04 and python 3.6.

Maybe there're some ordering/sorting differences between
python2 and python3. Or is somehow uses a different
hashing algorithm.

metze


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20181212/85104855/signature.sig>