[distcc] PCH Headers and distcc (again!)

[Fergus Henderson]

In response to Jamie Kirkpatrick and Sascha Demetrio's thread on precompiled
headers, I wrote the following reply in private email.  Now that distcc 3.0
is public, I'm copying it back to the public list.
I wrote:

> You may be interested to know that Google has been doing a lot of work on
> distcc and we're about to make our work public. We have some improvements to
> distcc that eliminate the local preprocessing bottleneck and which may help
> a lot in the same sort of situations where PCH's help. Perhaps you would be
> interested in beta-testing it?
>

And Sascha replied:

> This really sounds interesting, and I'd really like to take a look at
> what you have. [...]
>
> The idea of sending the header files to the server sounds interesting,
> but this rises some questions. (I also played with the idea of
> transferring all required files to the server, but never implemented
> anything except for the PCH patch.
>
> For debug builds, the compiler needs to see the correct file path as-is on
> the client, which may be difficult/impossible to reproduce on the
> server. One way to fix this is to run the preprocessor (on the server)
> separately and then patch the preprocessed file (convert '# 123
> "/var/distcc/file-cache/a6df3d6.h"' back to '# 123 "E:/someSDK/whatever.h")
> .
>
> Another thing: what happens with all the preprocessor related options
> (e.g. -MF). Running the preprocessor on the client machine solves this
> nicely, but when the preprocessor is run on the server, then these files
> must be patched and sent back to the client. Or the client (which scans
> the files anyway) interprets these options and generates the dependency
> files?
>
> Another (I think more elegant) way to deal with this is to run the
> compiler with an LD_PRELOAD that wraps
> open()/read()/access()/stat()/opendir()/readdir()/closedir(). Any source
> files opened read-only by the compiler could then be transferred
> on-demand to a local file cache. Write-only files are sent back to the
> client. (Some magic may be required to figure out which files to
> transmit and which files to use from the local compiler installation,
> but I am sure this can be sorted out.)
>
> The big advantages of an LD_PRELOAD based solution are:
>
>    1. Works with any compiler. No need to rely on GCC specific features.
>   2. Automagically works with PCHs.
>   3. Automatically recognizes all output files generated by the compiler.
>   4. Will work with non-code-compilation processes (e.g. compiling
>      documentation).
>
>  Handling directory traversals might require special attention in a
> LD_PRELOAD based solution. A possible way to deal with this: directory
> listings are sent to the server on demand, but the client keeps track of
> which directories are typically scanned by the compiler (maybe based on
> the location of the source file and the -I switches passed to the
> compiler). For subsequent compilation requests with the same -I options
> and source file directory, the contents of the directories scanned by
> the compiler are sent with the initial transfer to the server. This way
> the compiler can scan the include path without any additional delay.
>
>  As stated above, I have not tried this in practice - it just an "idea" :)
>

-- 
Fergus Henderson <fergus at google.com>
-------------- next part --------------
HTML attachment scrubbed and removed