[clug] aggregating disks across multiple machines

[Andrew Janke]

> 1. Staff should keep only working copies locally, the master should be
>   elsewhere, ideally in a VCS style "edit, then commit" arrangement.

Except in the case where the data is enormous. In what we do for
example I can generate 1TB in 24 hours without any sweat.

>> A much better solution would be aggregate disks into a single, large,
>> duplicated, data warehouse and have it accessible to all machines.
>
> ...maybe.  It depends a lot on how your data is used; given the size of the
> workstation I am *guessing* that your workload involves a lot of data pounding
> on the workstation.

+1

> Pushing that to a central machine implies that you need to deliver N * <the
> size of your group> random IOPS on that machine, instead of just N random IOPS
> per machine.
>
> That probably isn't cost-effective for you.

+1

> Almost certainly not, IMO.  The cost, in performance and complexity, is likely
> to overcome *any* benefit from a shared namespace you get.

In these cases given that a single user is usually on one machine I
set up a bunch of AutoFS cross mounts and put each users data on their
own machine (and backed up to another via rsync at night). Then if you
are running some sort of clustering/batch processing thing you get the
best of both worlds. I would be VERY cautious about using a
distributed file system as you only need one user to crash X when
npviewer bombs on some webpage or a kernel panic by some other means
and you have a hung filesystem across all the machines. Or perhaps
only one machines RAM/HDD to die and everything goes down.

So I define a mountpoint called /giglo and set up the extra space
(locally mounted as /export{01,02,03}) as such:

murdoch:~$ cat /etc/exports
/export01   10.10.100.0/255.255.255.0(rw,sync,subtree_check)

murdoch:~$ cat /etc/auto.master
# /etc/auto.master

/giglo            /etc/AutoFS/auto.giglo     -fstype=nfs,hard,intr,nodev

murdoch:~$ cat /etc/AutoFS/auto.giglo
#! /usr/bin/perl
#
# /etc/AutoFS/auto.giglo
#
# Andrew Janke - a.janke at gmail.com
#
# put something like this in auto.master
# giglo    /etc/AutoFS/auto.giglo     -fstype=nfs,hard,intr,nodev

use strict;
use Socket;

# static vars
my $mount_options = "-fstype=nfs,hard,intr,nodev";
my $giglo_net = "10.10.100";
my $domain = "lclust";

my($input, $server, $drive_num, $server_ip, $last_digit);

my $input = shift(@ARGV);

# split the server and number
($server, $drive_num) = split(/\-/, $input, 2);

# add default drive (01) if missing
$drive_num = '01' if length($drive_num) == 0;

# reality check MKI
exit if length($drive_num) != 2;

# get and clean up the IP address of the current host
# chomp($server_ip = `dig +short $server.$domain`);
$server_ip = inet_ntoa(inet_aton($server));

# reality check MKII
exit if $server_ip eq "";

# output the resulting mount line
print "$mount_options\t$server_ip:/export$drive_num\n";

Note that you will have to modify the "giglo_net" to your own IP
address space. I use giglo as I tend to run a second gigabit local
network among the machines to segregate NFS and other traffic.

So then if you have two machines called bill and ben, on all machines
you can access their /export01 drive as such:

murdoch:~$ ls /giglo/ben-01

or

murdoch:~$ ls /giglo/ben-02

for /export02

So users can still write their scripts with consistent mountpoints and
AutoFS will sort things out on the local machine.


--
Andrew Janke
(a.janke at gmail.com || http://a.janke.googlepages.com/)
Canberra->Australia    +61 (402) 700 883