[distcc] Alternatives to -j parallel

[Richard Beare]

I'll have a look at that - perhaps ensuring LD is set correctly will be
enough

Thanks

On Tue, Sep 6, 2016 at 8:56 PM, Łukasz Tasz <lukasz at tasz.eu> wrote:

> I think I got your point,
>
> playing with MKFLAGS with cmake will be a tricky part and complex at the
> end.
> I would recommend to use distcc and limit --localslots=1 and assure that
> linking starts with distcc at the begining.
> Make sure that your LD variable is prefixed with distcc.
>
> In my opinion it will be the easiest way.
>
> In case of pure make it is very easy, enough is to call $MAKE MKFLAGS= and
> -j then is 1, but it is not a case when talking about cmake buildsystem.
>
> regards
> L.
>
> Łukasz Tasz
> RTKW
>
> 2016-09-06 11:47 GMT+02:00 Richard Beare <richard.beare at gmail.com>:
>
>> Thanks,
>> Here's another way of describing my problem.
>>
>> I have a configuration that allows me to substitute distcc for cc/cxx.
>> This works fine, and I can distribute compilation across the 4 nodes
>> available with circleCI, when I supply a -j N option to make.
>>
>> However, it appears that the linking (ld) phase is carried out using a
>> separate mechanism - i.e. not from distcc. Thus the requirement of only one
>> job on the localhost is not respected by the link phase because distcc
>> isn't executing the linker. In the circleCI system this leads to problems
>> with memory.
>>
>> I'd therefore like to be able to tell distcc to be parallel without
>> telling the rest of the makefile also being parallel. Obviously this would
>> require that there is some other mechanism (besides -j) - perhaps an
>> environment variable - that triggers parallel behaviour in distcc.
>>
>> In short - the cc part of distcc works fine. However, in this build
>> system, ld is not being run from distcc. I don't want multiple ld jobs
>> running, so I'd like to be able to run make -j1 but instruct distcc to use
>> a higher level of j.
>>
>>
>>
>>
>> On Tue, Sep 6, 2016 at 6:42 PM, Łukasz Tasz <lukasz at tasz.eu> wrote:
>>
>>> Could you please describe more what you would like to achieve?
>>> -j does net decide it compilation will be done with distcc or not,
>>> -j must take into consideration how many distcc agent you have.
>>>
>>> you have two dimensions, one is if distcc should be used, and second is
>>> about parallelism.
>>> first dimension is controlled in build system, second can be controlled
>>> by build system, a but also may be limited by distcc.
>>>
>>> localslots and localslots_cpp controls parallelism from distcc point of
>>> view.
>>>
>>> regards
>>> Lukasz
>>>
>>> Łukasz Tasz
>>> RTKW
>>>
>>> 2016-09-06 4:11 GMT+02:00 Richard Beare <richard.beare at gmail.com>:
>>>
>>>> Hi,
>>>> I'm having a few problems using distcc for a large project on the
>>>> circleCI infrastructure, which has 4GB limits for the compilation hosts.
>>>>
>>>> Occasionally I see this limit being exceeded when 2 link jobs run on
>>>> the main server at the same time. This happens despite the --localslots=1
>>>> and localhost/1 settings in the distcc hosts file.
>>>>
>>>> I suspect the problem is that the large and complex cmake project that
>>>> is being built has its own link mechanism. So my question is whether there
>>>> is an alternative way to pass the parallel option to distcc besides make
>>>> -jN? This would allow the other compile steps that might not be under
>>>> distcc control to run serially on the local host.
>>>>
>>>> Thanks
>>>>
>>>> __
>>>> distcc mailing list            http://distcc.samba.org/
>>>> To unsubscribe or change options:
>>>> https://lists.samba.org/mailman/listinfo/distcc
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20160906/6628f7e6/attachment.html>