[clug] Basic work queue

[Adam Thomas]

On 18 June 2011 17:19, Carlo Hamalainen <carlo.hamalainen at gmail.com> wrote:
> Hi,
>
> I have a long computation that I would like to run on Amazon EC2. I
> want to manually start up a few high-CPU instances (say 5 or 10) and
> have each instance request a task from some master instance, do the
> task, signal completion to the master, and then get the next available
> task.
>
> The problem is "embarrassingly parallel":
>
> * The master only needs to send a 100 line shell script to an instance
> to get it running on a new task.
> * No instances need to communicate with each other.
>
> What is the simplest way to make such a work queue?

I think gearman[0] might do what you're looking for. I've played with
it a bit and generally find it very easy to setup and get going.

[0]: http://gearman.org/

>
> I did look at Amazon SQS [1] but it says that the queue messages can
> live for only up to 14 days, but my queue might last for a month or
> more, depending on how many worker instances I have running.
>
> Is something like RabbitMQ suitable?
>
> Would it be easier to set up PBS and use my Linode as the master?
>
> [1] http://aws.amazon.com/sqs/
>
> --
> Carlo Hamalainen
> http://carlo-hamalainen.net
> --
> linux mailing list
> linux at lists.samba.org
> https://lists.samba.org/mailman/listinfo/linux
>