[distcc] include_server macro evaluation

[Alastair Rankine]

Hi, I've been spending a bit of time with the distcc include_server trying to get a handle on issue #16. I'm making progress but not quite got a fix just yet. Here's what I've got so far.

Firstly I think the ## substitution in macro_eval.py is not quite right because it doesn't support whitespace around the ## operator. This is used in boost and is at least part of the problem with issue #16. Fortunately it's an easy fix:


=== modified file 'include_server/macro_eval.py'
--- include_server/macro_eval.py	2009-01-30 17:27:30 +0000
+++ include_server/macro_eval.py	2011-02-13 10:23:12 +0000
@@ -142,7 +142,7 @@
 # REGULAR EXPRESSIONS
 
 SINGLE_POUND_RE = re.compile(r"\B#\s*(\S*)") # \B = here: not at end of word
-DOUBLE_POUND_RE = re.compile(r"##")
+DOUBLE_POUND_RE = re.compile(r"\s*##\s*")
 SYMBOL_RE = re.compile(r"\b\w+\b") # \b = word boundary \w = word constituent
 

Secondly I want to question the decision to include the unsubstituted macro call in the set of possible values for expansion. I understand the justification for this for non-function-like macros, and for the preprocessor in general, but for the limited cases of macro expansion that the include_server needs (ie pretty much just calculated includes), it doesn't seem to add a lot of value.

I guess in theory there could be include files which look like macro invocations to the parser but are in fact real files. I'm not sure if there are any real examples of this happening, but it seems to me the cost of handling these things could be quite high. At least, the proliferation of unexpanded macros into the value_set contributes to a combinatorial explosion which I believe is at the heart of issue #16.

In the interest of clarity, the following test case diff should explain what I think is correct behaviour from macro_eval.py:


=== modified file 'include_server/macro_eval_test.py'
--- include_server/macro_eval_test.py	2008-07-29 21:19:10 +0000
+++ include_server/macro_eval_test.py	2011-02-13 10:53:30 +0000
@@ -102,26 +102,33 @@
     self.assertEqual(
       macro_eval.EvalExpression("max(2, 4)",
 		      { 'max': [ ( ['x', 'y'], "x < y? y: x") ] }),
-      set(['max(2, 4)', '2 <  4?  4: 2']))
+      set(['2 <  4?  4: 2']))
 
     self.assertEqual(
       macro_eval.EvalExpression("F(2, 4)",
 		      { 'F': ['max'],
 			'max': [ ( ['x', 'y'], "x < y? y: x") ] }),
-      set(['max(2, 4)', 'F(2, 4)', '2 <  4?  4: 2']))
+      set(['2 <  4?  4: 2']))
 
     self.assertEqual(
       macro_eval.EvalExpression("max(max(1,2), 3)",
 		      { 'max': [ ( ['x', 'y'], "(x < y? y: x)") ] }),
-       set(['((1 < 2? 2: 1) <  3?  3: (1 < 2? 2: 1))',
-            'max(max(1,2), 3)',
-            '(max(1,2) <  3?  3: max(1,2))',
-            'max((1 < 2? 2: 1), 3)']))
+       set(['((1 < 2? 2: 1) <  3?  3: (1 < 2? 2: 1))']))
 
     self.assertEqual(



Hoping that there are still people interested in working on distcc, it seems to have gone a bit quiet of late!