[clug] Basic work queue

[Carlo Hamalainen]

Hi,

I have a long computation that I would like to run on Amazon EC2. I
want to manually start up a few high-CPU instances (say 5 or 10) and
have each instance request a task from some master instance, do the
task, signal completion to the master, and then get the next available
task.

The problem is "embarrassingly parallel":

* The master only needs to send a 100 line shell script to an instance
to get it running on a new task.
* No instances need to communicate with each other.

What is the simplest way to make such a work queue?

I did look at Amazon SQS [1] but it says that the queue messages can
live for only up to 14 days, but my queue might last for a month or
more, depending on how many worker instances I have running.

Is something like RabbitMQ suitable?

Would it be easier to set up PBS and use my Linode as the master?

[1] http://aws.amazon.com/sqs/

--
Carlo Hamalainen
http://carlo-hamalainen.net