Parallel serving via NFS from CTDB considered harmful

ronnie sahlberg ronniesahlberg at gmail.com
Wed Oct 24 21:35:55 MDT 2012


lock manager in nfs is broken by design and does not work reliably
even in the single-server case.
Just try tridges ping_pong tool and see how quickly it can get locking
to hang completely on a vanilla linux distro of your choice :-)

(nlm requires ordered and reliable delivery of udp packets for it to
work,  so it is pretty busted by design.
A stateless protocol that has sideeffects and that requires
indempotent behaviour and which does not even have replay protection
or ordering.
what were the fools at sun thinking?
)



This particular case with failover is not an issue with ctdb though
since it is careful to trigger a new graceperiod across the whole
cluster.
(still   nfs locking is still broken but that is per design/protocol
and nothing we can do much about :-( )



But to get back to your question :
File locking in NFSv2/3 is just unreliable and does not really work
that well. You should never rely on locking for nfs or you will be in
a world of pain.


regards
ronnie sahlberg

On Sat, Oct 20, 2012 at 4:43 AM, Jeff Layton <jlayton at samba.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I've been playing with CTDB recently and ran across a couple of pages
> that seem to indicate that serving a clustered filesystem from multiple
> nodes using NFS is safe:
>
>     http://ctdb.samba.org/nfs.html
>
> ...and...
>
>     https://wiki.samba.org/index.php/CTDB_Setup#Setting_up_CTDB_for_clustered_NFS
>
> I'm concerned that these pages are misleading since from what I can
> tell, when a failover event occurs the lock recovery grace period is not
> reinstated across the entire cluster.
>
> The problematic scenario is something like this:
>
> - - client1 mounts an exported filesystem from a public IP on serverA and
>   acquires a lock on it
>
> - - serverA crashes, its locks are released by DLM (or other clustered
> lock manager)
>
> - - client2 now races in and acquires a conflicting lock on serverB
>
> - - the ip address now "floats" to serverC
>
> - - client1 tries to reclaim his lock now, but can't
>
> ...even worse is a variant of the above case where client2 just briefly
> gets the lock, modifies the data protected by it and then releases it.
> client1 will think he's had exclusive access to the lock the whole
> time. The problem in that case will manifest itself as silent data
> corruption.
>
> Getting clustered NFS locking right is really, really hard due to the
> recovery semantics. I'd like to suggest that we either remove the pages
> above, or at least add some warnings that locking may not be reliable
> in such configurations.
>
> Thoughts?
> - --
> Jeff Layton <jlayton at samba.org>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.18 (GNU/Linux)
>
> iQIcBAEBAgAGBQJQgo5wAAoJEAAOaEEZVoIVaBsP/1JdwzKqxUbm4zATkHdhfqbQ
> MoZpdFQo6gxiWzJaQ1EkqPQoW20T9tOqtPWYZKiwe5OyX64Gvzpb7WSKleiiWFcE
> TGTFEwJGzkplikwDt6FN3G4RuGm8zyPGKRbIKEEjGeaJ2768WEkOkmV3L1WMcCN+
> xupCIFi4ukBQDfjrgNyrVYVAM1Qwe2SIMeN9YOQb+Ij0KgXTmXGoAMMsLe5QBv5I
> q1bNOfqlMRQ/Hu/mOoxY5IG7q3hPZL7eV8bTmwx0osZ8iqBLz2JNAQSYTJvXX5Bu
> m1MpgHH9rsEr1lsGnyLV4fMqo38laWbcilWR2YBY44wXzVMcaEJrbvG59KmqVT0h
> W9IjhRUn5PJK9K7vLcQ9bWcHyGdFFnzMqzfDTp+KBBrpcv2lc/FSy3/zpVKGG/7X
> 7Wj6LsdFGdNbvDuRdqokqyz8wXfwUWxJY3RVmOhNe7K9sNdccXVsDNfFztVTchWW
> wguZN8EfGV6mHtKbv4mCPNvyjTLHXL7wxCRHhMeTRK0E4aYMcePDiyooImmwMrDU
> VclxRq23x1HYWHjT0Qv+BmNUBRA3BcwwrxyptouDUV3lypzYLtcpukSCKULBUqkt
> tdy6q+5id+r4w/8A2NMda3v7aFDiVKzp0LB23gpEnu8bcZmoMGLIq+KBE5Yhiy20
> 7sVtUaj+dvPzaXRFz0xg
> =cbZn
> -----END PGP SIGNATURE-----


More information about the samba-technical mailing list