[distcc] PCH support patch for distcc

[Sascha Demetrio]

Hi!

The attached patch adds support for using precompiled headers (PCHs)
with distcc distributed builds. The patch is not ready for end-user
consumption, as it includes no documentation on how it works (except for
comments in the code).


IMPORTANT: If you plan to use this patch, you'll also need the
dcc_unlock() patch I posted to this list a few hours ago.


The protocol has been changed to version 3. With this change, the client
sends an additional word containing a set of flags. In these flags, the
client specifies if file transfer compression should be enabled and if
the preprocessed file refers to a PCH. If both flags are clear, the
protocol after that remains unchanged,

PCH support for the server is enabled by adding the ,pch option to the
host specification in DISTCC_HOSTS. This option may be combined with the
,lzo option.

The client PCH support is enabled either if the compiler command line
contains the option '-fpch-preprocess' or if the DISTCC_PCH environment
variable is set to non-zero.

If server and client have the same compiler binary installed, then the
PCH patch works like this:

The preprocessed output file is scanned for '#pragma GCC pch_preprocess
"filename"'. In this line, the quoted file name is substituted by the
hex representation of the PCH file's MD5 hash. The client keeps a
pchfile+mtime to MD5-hash mapping in the ~/.distcc/state/pch file.

After receiving the preprocessed file, the server has a chance to query
the PCH file. It will do so if the PCH file is not already in a PCH file
cache. In case of a cache hit, the server simply sends a 'no thanks'
message to the client and starts the compilation. The server looks for
the PCH #pragma and replaces the MD5 hash with the path to the locally
cached PCH file.


Things get a bit more complicated if client and server run incompatible
GCC binaries. This is handled by compiling the PCH files on the server
and creating a local PCH proxy file. Here's the details:

If the DISTCC_PCH_REMOTE variable is set to non-zero, then the client
will recognize the header file as an input file and send it to the
server for remote pre-compilation. A PCH file contains information about
the preprocessor state, so the file sent to the server must include this
state information. This is done by setting the -dD option when the
header is being preprocessed. All #define's and #undef's are moved to
the end of the preprocessed file (except for built-in macros and macros
defined on the command line). This is required to make sure that the
macros do not interfere with the preprocessed file contents (which will
have all macros expanded) - double expansion would lead to incorrect
results in some cases.

The DISTCC_PCH_REMOTE option is useful even if server and client share
the same compiler binary, because it offloads the PCH pre-compilation
from the client to the server. That's the rationale for providing a
separate distcc option for that.

Limitation: The header file is recognized as an input file only if the
DISTCC_PCH_REMOTE option is set _and_ the header file immediately
follows the -c switch.

For PCHs to work, the local preprocessor must be able to extract the
preprocessor state from the PCH - which is not possible if the PCH file
has been created by a different compiler binary. To solve this, a local
proxy file is created by setting DISTCC_PCH_PROXY to non-zero.

The proxy file is created at the same location as the PCH file, with the
.gch suffix stripped. To prevent accidental clobbering of a real source
file, this is done only if that file does not exist, so a make rule for
creating the PCH file should remove the proxy file before starting the
remote PCH pre-compilation.

The proxy file contains the '#pragma GCC pch_preprocess "file"' line
followed by all #define's and #undef's captured from the PCH
preprocessing. The build environment should be set up in such a way that
the directory containing the proxy file is scanned before the directory
containing the real file. Obviously, the compiler should be caller
_without_ the -fpch-preprocess option, because the preprocessor should
leave the local PCH file alone.

Finally, the DISTCC_PCH option must be set to non-zero when compiling -
there's no -fpch-preprocess option in the command line, so distcc can
not automatically recognize that we're doing PCHs.


This PCH patch is in everyday use here on site and works pretty well so
far. BUT: The change is very intrusive and probably (certainly) contains
bugs. There has been little review of the code, so expect things to be
broken.


Kind regards,
Sascha Demetrio

___
This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Crytek GmbH - http://www.crytek.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: distcc-2.18.3-pch.diff.bz2
Type: application/octet-stream
Size: 22036 bytes
Desc: not available
Url : http://lists.samba.org/archive/distcc/attachments/20080604/caa48a15/distcc-2.18.3-pch.diff-0001.obj