[distcc] Alternatives to -j parallel

[Richard Beare]

Thanks,
Here's another way of describing my problem.

I have a configuration that allows me to substitute distcc for cc/cxx. This
works fine, and I can distribute compilation across the 4 nodes available
with circleCI, when I supply a -j N option to make.

However, it appears that the linking (ld) phase is carried out using a
separate mechanism - i.e. not from distcc. Thus the requirement of only one
job on the localhost is not respected by the link phase because distcc
isn't executing the linker. In the circleCI system this leads to problems
with memory.

I'd therefore like to be able to tell distcc to be parallel without telling
the rest of the makefile also being parallel. Obviously this would require
that there is some other mechanism (besides -j) - perhaps an environment
variable - that triggers parallel behaviour in distcc.

In short - the cc part of distcc works fine. However, in this build system,
ld is not being run from distcc. I don't want multiple ld jobs running, so
I'd like to be able to run make -j1 but instruct distcc to use a higher
level of j.




On Tue, Sep 6, 2016 at 6:42 PM, Łukasz Tasz <lukasz at tasz.eu> wrote:

> Could you please describe more what you would like to achieve?
> -j does net decide it compilation will be done with distcc or not,
> -j must take into consideration how many distcc agent you have.
>
> you have two dimensions, one is if distcc should be used, and second is
> about parallelism.
> first dimension is controlled in build system, second can be controlled by
> build system, a but also may be limited by distcc.
>
> localslots and localslots_cpp controls parallelism from distcc point of
> view.
>
> regards
> Lukasz
>
> Łukasz Tasz
> RTKW
>
> 2016-09-06 4:11 GMT+02:00 Richard Beare <richard.beare at gmail.com>:
>
>> Hi,
>> I'm having a few problems using distcc for a large project on the
>> circleCI infrastructure, which has 4GB limits for the compilation hosts.
>>
>> Occasionally I see this limit being exceeded when 2 link jobs run on the
>> main server at the same time. This happens despite the --localslots=1 and
>> localhost/1 settings in the distcc hosts file.
>>
>> I suspect the problem is that the large and complex cmake project that is
>> being built has its own link mechanism. So my question is whether there is
>> an alternative way to pass the parallel option to distcc besides make -jN?
>> This would allow the other compile steps that might not be under distcc
>> control to run serially on the local host.
>>
>> Thanks
>>
>> __
>> distcc mailing list            http://distcc.samba.org/
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/distcc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20160906/af745f41/attachment.html>