[distcc] distcc compilers

[Martin Pool]

On 30 Apr 2003, Enno Rehling <ennor at funcom.com> wrote:
> I've only used distcc with gcc and g++ so far (and never looked at the 
> code, so the following question may sound ignorant to people who have). Is 
> it possible to use it with other compilers?

Yes, it is in theory possible.  If their command line differs from
gcc, then you may need to change the bit of distcc that deals with
it.  

I think people have tried supporting Intel cc, but it has some
discrepancies with gcc in how it interprets the command line.

I should say that I don't intend to spend my personal time supporting
compilers that are not free.  But if people send reasonably clean
patches I will look at them.
 
> Or, to take this from a different angle: If I were to write my own custom 
> tool for whatever purpose, and wanted to parallelize it, could I use 
> distcc, and what kind of interface would the tool need to have? Is that a 
> crazy idea, or something people actually do?

Yes, I have been thinking and talking offline to some people about
refactoring distcc into a generic 'distexec' tool, plus a gcc-specific
'skin'.

I don't want to go all the way to a completely transparent task
migration system like MOSIX, with IO redirection, etc.  MOSIX already
exists.

However I think there might be benefit in a tool that is half a level
higher than rsh or ssh.  It might be designed like this:

  Generic level:

    - distributes load across machines
    - passes commands, input, output, and errors across the network
    - handles networking

  Skin:

    - understand command lines
    - knows whether jobs can be distributed or not
    - works out what the input and output files are
    - does any local pre or post processing
    - produces the remote command

I think the skin can be done only on the client side.

I need to change the protocol to support better scheduling and
compression.  When I do that I will try to keep cc-specific
assumptions out of the new one.  The change should not be too large.

So the protocol needs to change thus:

 - perhaps allow for more than one input or output file
 - no server-side alteration or parsing of the command
 - client tells the server where to substitute the temporary filenames
 - ...?

Possibly the protocol ought to allow for streaming in input or
output, but I'm not sure the complexity is justified.

If you have specific applications in mind then please post a
description so we can make sure a refactored distexec can handle them.

-- 
Martin