[clug] Playing funny with make..

[Daniel Pittman]

Andrew Janke <a.janke at gmail.com> writes:

>> On the silly suggestions front, though, install condor[1] and use DAGman to
>> schedule the job dependencies[2].  That makes it easy to run and retry jobs
>> appropriately, as well as to distribute the work over multiple machines
>> when you get sick of waiting on your own machine to do it all.
>
> Well I already use Sun Grid Engine on a daily basis for all heavy
> lifting. FWIW I have always failed to see why things like dependencies are
> so hard in Condor. In SGE all I do is this:
>
>    $ qsub -hold_jid <blah> ...
>
> Where <blah> can be either a job id or a regex to match job names.

*nod*  The dependency stuff does seem a bit baroque.  OTOH, SGE didn't seem to
have good support for Windows, and especially running jobs that require a
Windows Station and Desktop allocated, simultaneously, which is why I will
shortly find out just how painful Condor is in that regard.

> Still this doesn't solve my requirements.

I did prefix it with "silly suggestion", because I didn't think it would. ;)

[...]

> Given that SGE also supports qmake (distributed make), it would seem obvious
> to add a great big Makefile to the whole thing as then you can either type
> make for local execution or qmake for cluster execution.

*nod* That seems a fine choice.  The chained rules in make should be
reasonably portable, too, which may be important since I don't know what qmake
is built on, but GNU Make doesn't seem the prime candidate.

[...]

> Mind you there are tons of other libraries out there that a supposed to
> handle all this (perl libraries, Scons, Ruby, etc, etc) but they all have
> their own subtle nuances and tend to not be as cross platform as make and
> sh.

*nod*  Pretty much none of the other compilation or batching tools I have used
have been worth much of anything for larger jobs.  Not even the very small
definition of "larger" that I have used in practice, compared to you.

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons
   Looking for work?  Love Perl?  In Melbourne, Australia?  We are hiring.