dbwrap_py: rapid dbwrap backend prototyping and FoundationDB
jra at samba.org
Tue Jan 17 20:17:06 UTC 2023
On Tue, Jan 17, 2023 at 03:33:42PM +0100, Ralph Boehme wrote:
>I've hacked a new dbwrap backend called dbwrap_py that calls into
>Python for the database operations. This came out of the desire to
>play around with the many available distributed database that are
>available these days, eg Cassandra, etcd, FoundationDB and so on .
>All of them have Python bindings and implementing the required
>primitives in Python is much simpler then doing it in C.
>I've developed dbwrap_py hand in hand with an initial backend
>targetting FoundationDB which is a scalable, high-performance
>key-value store with strong consistency and transaction support
>Thanks to the transaction support in FoundationDB it is possible to
>implement locking using a dedicated locking record per record combined
>with a fencing token that guards modfying database operations. Other
>databases may offer other primitives to implement record locking.
>The whole thing is now functional enough such that it passes certain
>torture tests when configured to put locking.tdb and
>smbXsrv_open_global.tdb behind dbwrap_py.
>Currently the whole dbwrap layering runs unchanged on top of the
>backend. As locking is implemented in the backend and FoundationDB
>support watches, it would be possible to not use the current g_lock
>and dbwrap_watch backend for dbs enabled for FoundationDB.
>With metze's new smb2.bench.path-contention-shared benchmark I'm
>getting 25 opens/closes per second on my laptop with FoundationDB
>being a single node cluster running on the laptop as well:
>$ bin/smbtorture //localhost/test -U slow%Passw0rd
>Opened 4 connections with qdepth=1 => 4 loops
>Running for 10 seconds
>$ bin/smbtorture3 //localhost/test -U slow%Passw0rd LOCAL-DBWRAP-PY
>dbwrap_fetch_locked() took: 2799 us
>dbwrap_record_store() took: 4205 us
>dbwrap_exists() took: 364 us
>dbwrap_parse_record() took: 3571 us
>dbwrap_do_locked() took: 5552 us
>The code is here:
> dbwrap_py:module = samba.samba3.dbwrap_py_fdb
> dbwrap_py:locking.tdb = yes
> dbwrap_py:smbXsrv_open_global.tdb = yes
> dbwrap_py:test_db_py.tdb = yes
>FoundationDB would be a good candidate for a public cloud SMB frontend
>ala Azure SMB, this is not aiming at the ctdb high performance SMB NAS
>I'll try to get a small FoundationDB cluster setup in K8 to see how
>that feels like. Stay tune... ;)
Great work Ralph ! I'm looking forward to playing with this.
More information about the samba-technical