[Samba] Can't restore from GPFS snapshots, disk_free error

[Jonathan Buzzard]

On Thu, 2013-10-17 at 09:16 -0400, Eric Chris Garrison wrote:
> Hello,
> 
> We're trying to set up a GPFS system with Samba running on top with CTDB
> managing it.
> 
> I have snapshots set up to be accessible in every directory as the invisible
> directory .snap
> 
> The snapshots are in the following format:
> 
> /usr/lpp/mmfs/bin/mmcrsnapshot 1MB `TZ=GMT date + at GMT-%Y.%m.%d-%H.%M.%S`
> 
> Šand look like this from the UNIX level:
> 
> ~ecgarris/RSFS/.snap/@GMT-2013.10.16-20.00.01/thing1
> 
> I've set up shares like this:
> 
> [homes] 
> path = %H/RSFS
>         comment = RSFS Home Directories
>         browseable = No
>         shadow:snapdir = .snap
> #        shadow:basedir = %H/RSFS
>         shadow:fixinodes = yes
>         shadow:snapdirseverywhere=yes
> 

[SNIP]

> I'd appreciate any help. It feels like we're really close. Thanks.
> 

I am assuming that you are loading the shadow_copy2 and gpfs VFS
modules. Assuming that you don't have independent filesets as well then
scrap the snapshot directories all over the place and in the general
configuration do

# enable shadow copies
	shadow : snapdir = /gpfs/.snapshots
	shadow : basedir = /gpfs
	shadow : fixinodes = yes

Works for certain with GPFS 3.4.x and Samba 3.5.x and 3.6.x

Now the bad news, give it up anyway as snapshots are unworkable on a
GPFS files system that is in production. The following commands in my
experience should only be run under very light load (aka maintenance
window) on a production file system mmcrsnapshot, mmdelsnapshot and
mmunlinkfileset.

I have a wonderful Perl script that can be called from all nodes in the
file system able to run admin commands; will create snapshots as
required, will remove old snapshots as required, complete with full
locking (the script is called from crontab on multiple nodes so a node
down for maintenance does not cause you to loose snapshots) and it all
works beautifully in test. Runs for months without a problem.

Put it on a production system, and within days if not hours you will get
a total freeze on the file system when it blocks trying to get a
quiescent lock to either take or remove a snapshot.

If you wish I can send you a copy of my Perl script, my advice is forget
it as ahead only pain and anguish :-(


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.