[PATCH][WIP] Make the Samba AD DC multi-process

Mon Nov 28 04:05:09 UTC 2016

On Mon, 2016-11-21 at 17:01 +1300, Andrew Bartlett wrote:
> On Fri, 2016-10-14 at 20:01 +1300, Andrew Bartlett wrote:
> > G'Day,
> > 
> > Attached is a WIP set of patches to make Samba connect to ldb
> > faster,
> > for the @ATTRIBUTES load, used until we can read the full schema.
> > 
> > This avoids some O(n^2) behaviour for the 600 attributes in the
> > default
> > schema, which was taking 5% of the time to run a simple ldbsearch.
> > 
> > Please consider, but don't push until I run the beachmarks.
> 
> Attached is some work in progress to do this, and to allow the LDAP
> and
> NETLOGON server to be multi-process. 
> 
> This breaks the RPC protocol by not checking the assoc_group when we
> accept a bind to the NETLOGON rpc server in the AD DC.
> 
> It also breaks the current link between the lsarpc services and
> netlogon, which currently allow a bind on either pipe to access these
> services. 
> 
> I've tried to make that all as generic as possible.
> 
> Please comment.  I don't plan to push this without coming back to the
> list. 
> 
> To address the cost of the additional processes, particularly for
> LDAP,
> we hope to have some improvments to talloc() to make short-lived
> child
> processes less costly at exit() time. 

I would appreciate comments on the attached patch set.  It includes the
foreshadowed improvements to talloc.

I would like to get at least the non-controversial parts of this into
Samba soon.  

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-perf-Add-simple-tests-for-the-open-close-a-database-.patch
Type: text/x-patch
Size: 5101 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0001-perf-Add-simple-tests-for-the-open-close-a-database--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-ldap-Run-the-LDAP-server-with-the-default-typically-.patch
Type: text/x-patch
Size: 1421 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0002-ldap-Run-the-LDAP-server-with-the-default-typically--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-ldb-Add-helper-function-ldb_schema_attribute_remove_.patch
Type: text/x-patch
Size: 3064 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0003-ldb-Add-helper-function-ldb_schema_attribute_remove_-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-ldb-Reduce-scope-of-allocation-and-de-allocation-of-.patch
Type: text/x-patch
Size: 5397 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0004-ldb-Reduce-scope-of-allocation-and-de-allocation-of--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-ldb-Reduce-per-attribute-memory-allocation-during-AT.patch
Type: text/x-patch
Size: 1146 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0005-ldb-Reduce-per-attribute-memory-allocation-during-AT-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-ldb-Add-helper-function-ldb_schema_attribute_fill_wi.patch
Type: text/x-patch
Size: 2618 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0006-ldb-Add-helper-function-ldb_schema_attribute_fill_wi-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-ldb-load-ATTRIBUTES-faster-by-sorting-once-not-at-ea.patch
Type: text/x-patch
Size: 5209 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0007-ldb-load-ATTRIBUTES-faster-by-sorting-once-not-at-ea-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-ldb-Bump-ABI-for-new-ldb-version.patch
Type: text/x-patch
Size: 23089 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0008-ldb-Bump-ABI-for-new-ldb-version-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-s4-rpc_server-Show-what-RPC-interfaces-are-listening.patch
Type: text/x-patch
Size: 1146 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0009-s4-rpc_server-Show-what-RPC-interfaces-are-listening-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-dsdb-specify-attributes-when-loading-schema.patch
Type: text/x-patch
Size: 3746 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0010-dsdb-specify-attributes-when-loading-schema-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-ldb-Avoid-individual-memory-allocations-when-searchi.patch
Type: text/x-patch
Size: 1150 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0011-ldb-Avoid-individual-memory-allocations-when-searchi-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0012-torture-Remove-access-to-LSARPC-via-pipe-netlogon-in.patch
Type: text/x-patch
Size: 4220 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0012-torture-Remove-access-to-LSARPC-via-pipe-netlogon-in-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0013-idl-Do-not-listen-for-lsarpc-on-pipe-netlogon.patch
Type: text/x-patch
Size: 1717 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0013-idl-Do-not-listen-for-lsarpc-on-pipe-netlogon-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0014-rpc_server-netlogon-Move-from-memcache-to-a-tdb-cach.patch
Type: text/x-patch
Size: 18692 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0014-rpc_server-netlogon-Move-from-memcache-to-a-tdb-cach-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0015-s4-rpc_server-Allow-each-interface-to-declare-if-it-.patch
Type: text/x-patch
Size: 4480 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0015-s4-rpc_server-Allow-each-interface-to-declare-if-it--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0016-s4-rpc_server-Allow-listener-for-RPC-servers-to-use-.patch
Type: text/x-patch
Size: 10532 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0016-s4-rpc_server-Allow-listener-for-RPC-servers-to-use--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0017-s4-rpc_server-Do-not-check-association-groups-for-NE.patch
Type: text/x-patch
Size: 3018 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0017-s4-rpc_server-Do-not-check-association-groups-for-NE-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0018-selftest-Use-rpc-server-port-netlogon-smb.conf-optio.patch
Type: text/x-patch
Size: 994 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0018-selftest-Use-rpc-server-port-netlogon-smb.conf-optio-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0019-s4-netlogon-Push-the-netlogon-server-in-the-AD-DC-in.patch
Type: text/x-patch
Size: 1525 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0019-s4-netlogon-Push-the-netlogon-server-in-the-AD-DC-in-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0020-s4-rpc_server-Avoid-extern-reference-to-dcesrv_mgmt_.patch
Type: text/x-patch
Size: 2032 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0020-s4-rpc_server-Avoid-extern-reference-to-dcesrv_mgmt_-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0021-pidl-Make-dcesrv-_-name-_interface-static-const.patch
Type: text/x-patch
Size: 1031 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0021-pidl-Make-dcesrv-_-name-_interface-static-const-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0022-pidl-Use-a-static-const-initialised-struct-in-dcerpc.patch
Type: text/x-patch
Size: 1556 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0022-pidl-Use-a-static-const-initialised-struct-in-dcerpc-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0023-s4-rpc_server-Use-a-type-safe-struct-signature-in-dc.patch
Type: text/x-patch
Size: 2084 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0023-s4-rpc_server-Use-a-type-safe-struct-signature-in-dc-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0024-s4-rpc_server-Use-an-initialised-static-const-struct.patch
Type: text/x-patch
Size: 1568 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0024-s4-rpc_server-Use-an-initialised-static-const-struct-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0025-pidl-Change-_get_pipe_fns-to-return-const-struct-api.patch
Type: text/x-patch
Size: 1944 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0025-pidl-Change-_get_pipe_fns-to-return-const-struct-api-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0026-pidl-Make-static-struct-api_struct-also-const.patch
Type: text/x-patch
Size: 876 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0026-pidl-Make-static-struct-api_struct-also-const-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0027-talloc-Improve-test_magic_protection-testsuite.patch
Type: text/x-patch
Size: 2302 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0027-talloc-Improve-test_magic_protection-testsuite-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0028-talloc-Add-tests-for-talloc_parent-after-realloc-of-.patch
Type: text/x-patch
Size: 1797 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0028-talloc-Add-tests-for-talloc_parent-after-realloc-of--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0029-talloc-Remove-talloc_abort_unknown_value-consolidate.patch
Type: text/x-patch
Size: 2026 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0029-talloc-Remove-talloc_abort_unknown_value-consolidate-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0030-talloc-clarify-that-talloc_magic-never-includes-the-.patch
Type: text/x-patch
Size: 975 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0030-talloc-clarify-that-talloc_magic-never-includes-the--0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0031-talloc-Set-TALLOC_FLAG_HAS_DESTRUCTOR-on-parents-whe.patch
Type: text/x-patch
Size: 2640 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0031-talloc-Set-TALLOC_FLAG_HAS_DESTRUCTOR-on-parents-whe-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0032-talloc-Only-call-talloc-destructors-when-TALLOC_FLAG.patch
Type: text/x-patch
Size: 5448 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0032-talloc-Only-call-talloc-destructors-when-TALLOC_FLAG-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0033-talloc-Add-debugging-abort-calls-for-invalid-situati.patch
Type: text/x-patch
Size: 1151 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0033-talloc-Add-debugging-abort-calls-for-invalid-situati-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0034-talloc-Pass-down-a-flag-to-determine-if-we-should-at.patch
Type: text/x-patch
Size: 3150 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0034-talloc-Pass-down-a-flag-to-determine-if-we-should-at-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0035-talloc-Avoid-calling-free-on-the-top-level-pointer-i.patch
Type: text/x-patch
Size: 1866 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0035-talloc-Avoid-calling-free-on-the-top-level-pointer-i-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0036-talloc-Add-talloc_free_for_exit.patch
Type: text/x-patch
Size: 14615 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0036-talloc-Add-talloc_free_for_exit-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0037-talloc-Use-talloc_free_for_exit-in-talloc_autofree.patch
Type: text/x-patch
Size: 1529 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0037-talloc-Use-talloc_free_for_exit-in-talloc_autofree-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0038-talloc-Add-tests-for-talloc-destructor-behaviour.patch
Type: text/x-patch
Size: 8058 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0038-talloc-Add-tests-for-talloc-destructor-behaviour-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0039-talloc-add-ASCII-art-to-describe-parent-child-arrang.patch
Type: text/x-patch
Size: 1330 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0039-talloc-add-ASCII-art-to-describe-parent-child-arrang-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0040-s4-process_standard-Use-talloc_free_for_exit.patch
Type: text/x-patch
Size: 996 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20161128/60902927/0040-s4-process_standard-Use-talloc_free_for_exit-0001.bin>