[distcc] PCH Headers and distcc (again!)

[Fergus Henderson]

Still more private email copied back to the public list.  This is the last
one, I promise :)

I <fergus at google.com> wrote:
>
> Ah, but then now you have all the complexity of include scanning, plus all
> the complexity of LD_PRELOAD.
> At this point I think this solution stops looking so elegant.  Since
> include scanning alone is sufficient to give
> good performance, why add that extra complexity?  Increased generality, I
> suppose, but since the performance
> is going to be poor for cases which include scanning doesn't catch, it's
> not clear that this is a significant win over
> just falling back to local compilation for those cases, and so it's not
> clear that the increased complexity is
> worthwhile.  IMHO.
>

Sascha replied:

> The complexity of this is the main reason why I did not implement it :)
>
> The situation I have here (which is probably a bit uncommon) is a lot of
> clients and a few powerful servers. All clients have a local checkout of
> the code base, so the idea of caching them on the server(s) comes
> naturally. Another plus of a preload+prescan solution is that it offers
> nice support for PCHs.
>
> Based on your pre-scan solution (if it really catches all required
> files), a much simpler LD_PRELOAD-based solution might be possible:
>
>   1. Client scans all files and compiles a list, computes all file
>      hashes, sends the hashes+file names to the server.
>   2. Server checks the cache and queries all missing files.
>   3. Client sends all queried files.
>   4. Server runs the compiler in a simple preload-wrapper, that simply
>      serves files from the cache. If the server scans include
>      directories, it will only see the files transferred by the client.
>      If a file is not found in the cache, the wrapper will try to open
>      the file on the server.
>
> Thanks for commenting my suggestion.
>

I then replied:

> The cacheing that you describe above is a great idea.
>
> But I'd rather rely on the include scanner being conservative (always
> over-approximating the set of needed files)
> than using an LD_PRELOAD to fetch missing files.
>

Sascha then replied:

> Agreed.
>
> At that point the preload is technically no longer required. I still see
> it as an option for translating the cached files to the file names as
> seen on the client (eliminating the need to patch the debug info in the
> generated ELF files) - but that's just an alternative to creating a
> /tmp/distcc/whatever directory containing links to the cached files.
> Such a preload wrapper would be really simple.
>
> One question remains - how do you handle nasty things like '#include
> MACRO(args)'?
>
> A possible way to handle problematic cases like this is to simply detect
> them in the pre-scan and fall back to a local preprocessor run with '-E
> -H -o /dev/null' in that case.
>
> My PCH patch already contains an file cache implementation for PCHs
> which could be generalized. I'll try to send the patch later today.
>

I then replied to the question "how do you handle nasty things like
'#include MACRO(args)'?":

The include scanner does a static analysis which gives a conservative
> over-approximation of the possible values that each macro can expand to.
>
> This is still not perfect, because the static analysis can take exponential
> time (or worse) in the worst case.
> So in cases where we detect that the static analysis is taking too long,
> the include server gives up, and the distcc client will then fall back to
> doing local preprocessing.


-- 
Fergus Henderson <fergus at google.com>
-------------- next part --------------
HTML attachment scrubbed and removed