[Samba] Can't restore from GPFS snapshots, disk_free error

[Jonathan Buzzard]

On 17/10/13 18:49, Eric Chris Garrison wrote:
> Thank you for the suggestions and advice, Jonathan.  We will try out
> the snapshot options you mention.
>
> How onerous are snapshots on a prod system?

When it goes wrong you are looking at file system where basically 
*nothing* happens till you restart the node that is trying to get a 
quiescent lock. Basically your file system is undergoing a denial of 
service.

The advice from IBM was basically "don't do that" unless the file system 
is quiet. Of course given you are running a Samba server and you don't 
control your end users who could of course be doing anything at anytime, 
and especially in a research environment (they keep funny hours) that is 
not exactly helpful.

> I think we're going to do
> just once a day, in the wee hours when not much is going on. What
> kind of problems do you see?

How does 07:00 on a Saturday morning grab you? I would have thought that 
it would have been pretty quiet at that point, but guess what...

My plan had been to take snapshots at 07:00, 10:00, 13:00 and 16:00 to 
give some roll back for files that got deleted/scrambled during the day 
which would otherwise not exist in the backup. Unless it has improved 
radically in GPFS 3.5 it is a no go in my experience.

> How badly does it hang, for how long?

Indefinitely, that is until the node attempting to get a quiescent lock 
is restarted. As you can imagine that is not funny.

> What sorts of things aggravate that?
>

Basically too much activity on the file system. Problem is you have 
absolutely no way of knowing in advance how much activity is going on.

It looked lovely on my test system. Ran for months without issue. Put in 
production we had lockup's within a couple of days. When in hung at 
07:00 on a Saturday morning I turned it off.

Note there are lots of gotchas running the GPFS/Samba/CTDB combination, 
and this is just one...


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.