[distcc] Re: PCH support patch for distcc

[Fergus Henderson]

Hi Sascha,

Thanks for the patch.  There are going to be some non-trivial issues merging
this patch in with the other changes in distcc 3.0, unfortunately.
For example, both distcc 3.0 and this patch add protocol version 3.

I'm hoping that the speed benefits of distcc 3.0 will help persuade you that
it's worth your effort to merge this into distcc 3.0 :)

Cheers,
  Fergus.

On Wed, Jun 4, 2008 at 12:20 PM, Sascha Demetrio <saschad at crytek.de> wrote:

> Hi!
>
> The attached patch adds support for using precompiled headers (PCHs)
> with distcc distributed builds. The patch is not ready for end-user
> consumption, as it includes no documentation on how it works (except for
> comments in the code).
>
>
> IMPORTANT: If you plan to use this patch, you'll also need the
> dcc_unlock() patch I posted to this list a few hours ago.
>
>
> The protocol has been changed to version 3. With this change, the client
> sends an additional word containing a set of flags. In these flags, the
> client specifies if file transfer compression should be enabled and if
> the preprocessed file refers to a PCH. If both flags are clear, the
> protocol after that remains unchanged,
>
> PCH support for the server is enabled by adding the ,pch option to the
> host specification in DISTCC_HOSTS. This option may be combined with the
> ,lzo option.
>
> The client PCH support is enabled either if the compiler command line
> contains the option '-fpch-preprocess' or if the DISTCC_PCH environment
> variable is set to non-zero.
>
> If server and client have the same compiler binary installed, then the
> PCH patch works like this:
>
> The preprocessed output file is scanned for '#pragma GCC pch_preprocess
> "filename"'. In this line, the quoted file name is substituted by the
> hex representation of the PCH file's MD5 hash. The client keeps a
> pchfile+mtime to MD5-hash mapping in the ~/.distcc/state/pch file.
>
> After receiving the preprocessed file, the server has a chance to query
> the PCH file. It will do so if the PCH file is not already in a PCH file
> cache. In case of a cache hit, the server simply sends a 'no thanks'
> message to the client and starts the compilation. The server looks for
> the PCH #pragma and replaces the MD5 hash with the path to the locally
> cached PCH file.
>
>
> Things get a bit more complicated if client and server run incompatible
> GCC binaries. This is handled by compiling the PCH files on the server
> and creating a local PCH proxy file. Here's the details:
>
> If the DISTCC_PCH_REMOTE variable is set to non-zero, then the client
> will recognize the header file as an input file and send it to the
> server for remote pre-compilation. A PCH file contains information about
> the preprocessor state, so the file sent to the server must include this
> state information. This is done by setting the -dD option when the
> header is being preprocessed. All #define's and #undef's are moved to
> the end of the preprocessed file (except for built-in macros and macros
> defined on the command line). This is required to make sure that the
> macros do not interfere with the preprocessed file contents (which will
> have all macros expanded) - double expansion would lead to incorrect
> results in some cases.
>
> The DISTCC_PCH_REMOTE option is useful even if server and client share
> the same compiler binary, because it offloads the PCH pre-compilation
> from the client to the server. That's the rationale for providing a
> separate distcc option for that.
>
> Limitation: The header file is recognized as an input file only if the
> DISTCC_PCH_REMOTE option is set _and_ the header file immediately
> follows the -c switch.
>
> For PCHs to work, the local preprocessor must be able to extract the
> preprocessor state from the PCH - which is not possible if the PCH file
> has been created by a different compiler binary. To solve this, a local
> proxy file is created by setting DISTCC_PCH_PROXY to non-zero.
>
> The proxy file is created at the same location as the PCH file, with the
> .gch suffix stripped. To prevent accidental clobbering of a real source
> file, this is done only if that file does not exist, so a make rule for
> creating the PCH file should remove the proxy file before starting the
> remote PCH pre-compilation.
>
> The proxy file contains the '#pragma GCC pch_preprocess "file"' line
> followed by all #define's and #undef's captured from the PCH
> preprocessing. The build environment should be set up in such a way that
> the directory containing the proxy file is scanned before the directory
> containing the real file. Obviously, the compiler should be caller
> _without_ the -fpch-preprocess option, because the preprocessor should
> leave the local PCH file alone.
>
> Finally, the DISTCC_PCH option must be set to non-zero when compiling -
> there's no -fpch-preprocess option in the command line, so distcc can
> not automatically recognize that we're doing PCHs.
>
>
> This PCH patch is in everyday use here on site and works pretty well so
> far. BUT: The change is very intrusive and probably (certainly) contains
> bugs. There has been little review of the code, so expect things to be
> broken.
>
>
> Kind regards,
> Sascha Demetrio
>
> ___
> This message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and delete
> this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secure or error-free as information could be intercepted, corrupted, lost,
> destroyed, arrive late or incomplete, or contain viruses. The sender
> therefore does not accept liability for any errors or omissions in the
> contents of this message, which arise as a result of e-mail transmission. If
> verification is required please request a hard-copy version. Crytek GmbH -
> http://www.crytek.com
>



-- 
Fergus Henderson <fergus at google.com>
-------------- next part --------------
HTML attachment scrubbed and removed