Patch to support Scalable CTDB

[Ralph Böhme]

On Tue, May 01, 2018 at 11:37:58AM +1000, Martin Schwenke via samba-technical wrote:
> Unfortunately the diagrams  at:
> 
>   https://wiki.samba.org/index.php/Samba_%26_Clustering#Finding_the_DMASTER
> 
> are wrong.  I have a new diagram but need to discuss with people
> whether the above should be kept as a historical document or whether I
> should update.

please keep the reference and create a slick new page with the correct
diagram. Bonus points if you add the text below. Either way, I'll owe you a pot
of best green tea! :)

> CTDB uses 2 (relatively :-) simple concepts for doing the distribution:
> 
> * DMASTER (or data master)
> 
>   This is the node that has the most recent copy of a record.
> 
>   The big question is: How can you find this DMASTER?  The answer is...
> 
> * LMASTER (or location master)
> 
>   This node always knows which node is DMASTER.
> 
>   The LMASTER for a record is calculated by hashing the record key and
>   then doing a modulo of the number of active, LMASTER-capable nodes
>   and then mapping this to a node number via the VNNMAP.
> 
> Let's say you have 3 nodes (A, B, C) and node A wants a
> particular record. Let's say that node B is the LMASTER for that
> record.
> 
> There are 3 cases, depending on which node is DMASTER:
> 
> * DMASTER is A
> 
>   smbd will find the record locally.  No migration is necessary.  The
>   LMASTER is not consulted.
> 
> * DMASTER is B
> 
>   A will ask B for the record.  B will notice that it is DMASTER and
>   will forward the record to A.  The record will be updated on both A
>   and B because the change of DMASTER must be recorded.
> 
> * DMASTER is C
> 
>   A will ask B for the record.  B will notice that it is not DMASTER
>   and forward the request to C.  C forwards the record to B, which
>   forwards it to A.  The record will be updated on A, B and C because
>   the change of DMASTER must be recorded.
> 
> You can now add nodes D, E, F, ... and they will not affect migration
> of the record (if there is no contention for the record from those
> additional nodes).
> 
> If there is heavy contention for a record then 2 different performance
> issues can occur:
> 
> * High hop count
> 
>   Before C gets the request from node B, C responds to a migration
>   request from another node and is no longer DMASTER for the record.
>   C must then forward the request back to the LMASTER. This can go on
>   for a while. CTDB logs this sort of behaviour and keeps statistics.
> 
> * Record migrated away before smbd gets it
> 
>   The record is successfully migrated to node A and ctdbd informs the
>   requesting smbd that the record is there.  However, before smbd can
>   grab the record, a request is processed to migrate the record to
>   another node.  smbd looks, notices that node A is not DMASTER and
>   must once again ask ctdbd to migrate the record.  smbd may log if
>   there are multiple attempts to migrate a record.

Thanks!
-slow

-- 
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/
GPG Key Fingerprint:           FAE2 C608 8A24 2520 51C5
                               59E4 AA1E 9B71 2639 9E46