[clug] Sun Grid Engine

[Daniel Pittman]

Andrew Janke <a.janke at gmail.com> writes:

>> So... dragging this to another topic, I presume that is the Sun Grid Engine.
>> Generally speaking, how do you find working with the Sun Grid Engine?  Our
>> target machines would be Debian Linux, either etch or lenny, at this point.
>
> Works and works well for batch and parallel processing. There is a bit of a
> black art to setting up queues and cluster resources to make things work
> well for your particular requirements but the default FIFO works well to get
> you going.

I think configuring any clustering or batch processing software is a bit of a
black art, especially because each deployment is going to have radically
different requirements. :)

[...]

>> Also, it looks like it runs, kind-of, on Windows, as long as we have Interix
>> in place.  Since we need Windows, and to drive Microsoft Office via OLE
>> Automation, does that work with SGE?
>
> Sort of...  Meaning you want to do batch processing on windows machines on
> Office things via OLE?

*nod*  One of the less pleasant requirements we have is to generate PowerPoint
slide sets from data we collect and process.  Given that generating this has
consumed three or four hours of CPU time in the past, and that the only
interface that can do it is driving PPT via OLE automation[1]...

We also want to integrate this with the rest of the cluster: almost all the
data processing and preparation can be done on real (read Linux ;) machines,
but we need to ship the data somewhere that Windows can see it, drive PPT, and
ship the results back somewhere else.

> Surely there is a M$ widget to do this?

Part of the desire is to have one application, which we drive from inside our
software on Linux, if possible.  Finding some Microsoft hosted batch
processing solution is the next step down the line when ^W if something
integrated fails.

> In the past I have tried many iterations to use up the spare CPU cycles on
> Windows machines in the department but this isn't quite the same thing.
> More detail required!

Well, hopefully the above answers some of this.  The Windows machines will be
dedicated, though, or at least some of them will.  The cycle scavenging parts
of Condor are actually less interesting to us than the cross-platform stuff,
the data placement integration, and the DAGMAN feature.

>> Finally, do you have anything to say on the topic of SGE vs Condor for this
>> sort of work?
>
> Well for what we do (Automated medical image analysis) SGE is the better
> choice.

*nod*  I presume that is just "better fit" and not "Condor does X awfully". ;)

After all, if we also need to X then it would be nice to know about it before
I invest too much time in trialing condor. ;)

     Daniel

Footnotes: 
[1]  Part of this comes from a requirement to generate reports that are,
     visually, exactly what the client specified.  By "specified" I mean, of
     course, "supplied a PPT file and said ‘make it like this’".

-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons
   Looking for work?  Love Perl?  In Melbourne, Australia?  We are hiring.