[distcc] Alternatives to -j parallel

[Łukasz Tasz]

I think I got your point,

playing with MKFLAGS with cmake will be a tricky part and complex at the
end.
I would recommend to use distcc and limit --localslots=1 and assure that
linking starts with distcc at the begining.
Make sure that your LD variable is prefixed with distcc.

In my opinion it will be the easiest way.

In case of pure make it is very easy, enough is to call $MAKE MKFLAGS= and
-j then is 1, but it is not a case when talking about cmake buildsystem.

regards
L.

Łukasz Tasz
RTKW

2016-09-06 11:47 GMT+02:00 Richard Beare <richard.beare at gmail.com>:

> Thanks,
> Here's another way of describing my problem.
>
> I have a configuration that allows me to substitute distcc for cc/cxx.
> This works fine, and I can distribute compilation across the 4 nodes
> available with circleCI, when I supply a -j N option to make.
>
> However, it appears that the linking (ld) phase is carried out using a
> separate mechanism - i.e. not from distcc. Thus the requirement of only one
> job on the localhost is not respected by the link phase because distcc
> isn't executing the linker. In the circleCI system this leads to problems
> with memory.
>
> I'd therefore like to be able to tell distcc to be parallel without
> telling the rest of the makefile also being parallel. Obviously this would
> require that there is some other mechanism (besides -j) - perhaps an
> environment variable - that triggers parallel behaviour in distcc.
>
> In short - the cc part of distcc works fine. However, in this build
> system, ld is not being run from distcc. I don't want multiple ld jobs
> running, so I'd like to be able to run make -j1 but instruct distcc to use
> a higher level of j.
>
>
>
>
> On Tue, Sep 6, 2016 at 6:42 PM, Łukasz Tasz <lukasz at tasz.eu> wrote:
>
>> Could you please describe more what you would like to achieve?
>> -j does net decide it compilation will be done with distcc or not,
>> -j must take into consideration how many distcc agent you have.
>>
>> you have two dimensions, one is if distcc should be used, and second is
>> about parallelism.
>> first dimension is controlled in build system, second can be controlled
>> by build system, a but also may be limited by distcc.
>>
>> localslots and localslots_cpp controls parallelism from distcc point of
>> view.
>>
>> regards
>> Lukasz
>>
>> Łukasz Tasz
>> RTKW
>>
>> 2016-09-06 4:11 GMT+02:00 Richard Beare <richard.beare at gmail.com>:
>>
>>> Hi,
>>> I'm having a few problems using distcc for a large project on the
>>> circleCI infrastructure, which has 4GB limits for the compilation hosts.
>>>
>>> Occasionally I see this limit being exceeded when 2 link jobs run on the
>>> main server at the same time. This happens despite the --localslots=1 and
>>> localhost/1 settings in the distcc hosts file.
>>>
>>> I suspect the problem is that the large and complex cmake project that
>>> is being built has its own link mechanism. So my question is whether there
>>> is an alternative way to pass the parallel option to distcc besides make
>>> -jN? This would allow the other compile steps that might not be under
>>> distcc control to run serially on the local host.
>>>
>>> Thanks
>>>
>>> __
>>> distcc mailing list            http://distcc.samba.org/
>>> To unsubscribe or change options:
>>> https://lists.samba.org/mailman/listinfo/distcc
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20160906/fe309b34/attachment-0001.html>