samba_dnsupdate timeouts (was Re: [PATCH] python indent bugfix in dns_hub.py)
Stefan Metzmacher
metze at samba.org
Mon Feb 11 08:17:55 UTC 2019
Am 07.02.19 um 15:54 schrieb Stefan Metzmacher via samba-technical:
> Am 07.02.19 um 10:37 schrieb Stefan Metzmacher via samba-technical:
>> Hi Tim,
>>
>>> I just wanted to say I think dns_hub has been a good addition to the
>>> selftest framework. Anything that allows us to test DNS more
>>> realistically is a good thing, and is worth a few teething problems.
>>
>> Exactly we should make more use of it and remove the need for the
>> RESOLV_WRAPPER_HOSTS file.
>>
>>> It at least highlighted the real problem, which was that we were
>>> starting to hit the CI runner limits, and that could've dragged on for
>>> months with CI failing intermittently for no obvious reason. For the
>>> record, the CI limit seems to be around 8 DCs, although obviously this
>>> varies somewhat depending on the process model overhead. Tearing down
>>> testenvs once we're done with them seems like a good idea in the long run.
>>
>> BTW: there's at least one additional problem I noticed while cleaning up
>> my patchset to wait for the dns_update_cache file to be filled.
>>
>> On an RODC samba_dnsupdate calls DsrUpdateReadOnlyServerDnsRecords via
>> IRPC to the local winbindd, which calls
>> DsrUpdateReadOnlyServerDnsRecords via netlogon to the RWDC.
>> The netlogon server calls dnsupdate_RODC via IRPC to the dnsupdate
>> task on the RWDC, which calls samba_dnsupdate with a temporary config
>> on behalf of the RODC.
>>
>> Currently samba_dnsupdate (on the RODC) constantly recreates its irpc
>> handle (and the messaging context and the dgram socket). This causes
>> problems when winbindd tries to send back the result to samba_dnsupdate,
>> as winbindd's messaging context caches connected dgram sockets per
>> target pid for 1 second. As the target (samba_dnsupdate) constantly
>> recreates its socket, winbindd very likely hits ECONNREFUSED when
>> the socket is recreated multiple times within 1 second.
>> As a result samba_dnsupdate hits a 10 second irpc timeout, so the
>> whole samba_dnsupdate hits the 20 second timeout on the RODC.
>>
>> The solution to this is to cache the irpc handle in samba_dnsupdate.
>>
>>> I'll raise a bug and backport the autobuild change to 4.10.
>>
>> I'm currently testing the attached additional patches here:
>> https://gitlab.com/samba-team/devel/samba/pipelines/46503335
>> and
>> https://gitlab.com/samba-team/devel/samba/pipelines/46503395
>
> The results look good, every thing passes, but the time reduction
> provided by
> https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=9f808d4e1e5d
> is only ~ 25 mins.
>
> samba_ad_dc_ntvfs takes 201 mins:
> https://gitlab.com/samba-team/devel/samba/-/jobs/157703665
> and samba only 27 mins:
> https://gitlab.com/samba-team/devel/samba/-/jobs/157703659
>
> Before we had samba with 227 mins:
> https://gitlab.com/samba-team/devel/samba/-/jobs/157703448
>
> A real reduction is provided by this commit:
> https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=173e69dd4f5e
> so that we only run ad_dc_ntvfs in the new environment
>
> Now we have samba with 83 mins:
> https://gitlab.com/samba-team/devel/samba/-/jobs/157804397
> and samba_dc_dc_ntvfs with a failure at 635/636 tests after 94 mins:
> https://gitlab.com/samba-team/devel/samba/-/jobs/157804407
>
> The failure is reproducable:
>
> [635(4196)/636 at 1h21m26s]
> samba4.blackbox.dbcheck(ad_dc_ntvfs)(ad_dc_ntvfs:local)
> UNEXPECTED(failure):
> samba4.blackbox.dbcheck(ad_dc_ntvfs).dbcheck(ad_dc_ntvfs:local)
> REASON: Exception: Exception: WARNING: The "lsa over netlogon" option is
> deprecated
> WARNING: The "server schannel" option is deprecated
> Checking 10192 objects
> NOTE: old (due to rename or delete) DN string component for
> defaultObjectCategory in object
> CN=schemaInfo-Class-1549544055-NEW,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com
> -
> CN=schemaInfo-Class-1549544055,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com
> Not fixing old string component
> NOTE: old (due to rename or delete) DN string component for
> lastKnownParent in object CN=Servers
>
> FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)
>
> A summary with detailed information can be found in:
> ./bin/ab/summary
>
> Or in a private autobuild:
>
> [635(4196)/636 at 1h41m17s]
> samba4.blackbox.dbcheck(ad_dc_ntvfs)(ad_dc_ntvfs:local)
> UNEXPECTED(failure):
> samba4.blackbox.dbcheck(ad_dc_ntvfs).dbcheck(ad_dc_ntvfs:local)
> REASON: Exception: Exception: WARNING: The "lsa over netlogon" option is
> deprecated
> WARNING: The "server schannel" option is deprecated
> Checking 10196 objects
> NOTE: old (due to rename or delete) DN string component for
> defaultObjectCategory in object
> CN=schemaInfo-Class-1549542279-NEW,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com
> -
> CN=schemaInfo-Class-1549542279,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com
> Not fixing old string component
> NOTE: old (due to rename or delete) DN string component for
> lastKnownParent in object CN=NTDS Settings
>
> FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)
>
> I'm currently running a private autobuild with
> autobuild-private.sh samba-ad-dc-ntvfs --nocleanup
> and try to debug it there.
>
> Any ideas why this fails when we remove rodc, vampire_dc and promoted_dc.
Somehow there were some normalization problems with msTSExpireDate4
attributes in the vlv test. 19000101010000.0Z is normalized to
19700101010000.0Z, so there's a problem with
dsdb_syntax_NTTIME_drsuapi_to_ldb() and
dsdb_syntax_NTTIME_ldb_to_drsuapi() loosing information.
This is a separate problem I'm ignoring for now, I've created
https://bugzilla.samba.org/show_bug.cgi?id=13783
I fixed the vlv test to correctly cleanup the created objects.
The NOTE: old (due to rename or delete) DN string component for...
warnings were supposed to be ignored, but Commit
35bfc62a31c9ad73449594ddd48f76f50e0abade changed
dbcheck to not regard old one-way-links as errors.
At that time the relavant trigger changed from
fix_all_string_dn_component_mismatch to
fix_all_old_dn_string_component_mismatch.
Here's a patchset that fixes these problems and reduces the
total time for autobuild and gitlab by ~55% down to ~ 100mins.
See also https://gitlab.com/samba-team/samba/merge_requests/251
Please review and push:-)
Thanks!
metze
-------------- next part --------------
From a0fc45f89572960bb927ef2b8db059af2f4b53cb Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Mon, 4 Feb 2019 09:28:07 +1300
Subject: [PATCH 1/5] selftest: Make dns_hub socket timeout match
DNS_REQUEST_TIMEOUT
I was hitting the recv_packet = s.recv(2048, 0) exception because
the socket timeout was reached. We've seen it before, but it seemed more
common after changing the default process-model to prefork. This patch
makes the socket timeout used by the python code consistent with the C
code.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
selftest/target/dns_hub.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/selftest/target/dns_hub.py b/selftest/target/dns_hub.py
index d2d1f39e752a..e0fe68e985f6 100755
--- a/selftest/target/dns_hub.py
+++ b/selftest/target/dns_hub.py
@@ -34,6 +34,8 @@ else:
import socketserver
sserver = socketserver
+DNS_REQUEST_TIMEOUT = 10
+
class DnsHandler(sserver.BaseRequestHandler):
def dns_transaction_udp(self, packet, host):
@@ -42,7 +44,7 @@ class DnsHandler(sserver.BaseRequestHandler):
try:
send_packet = ndr.ndr_pack(packet)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
- s.settimeout(5)
+ s.settimeout(DNS_REQUEST_TIMEOUT)
s.connect((host, 53))
s.sendall(send_packet, 0)
recv_packet = s.recv(2048, 0)
--
2.17.1
From b908207f454a27a3bba45b2c411434a0e3ed9fe4 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Wed, 23 Jan 2019 09:34:40 +0100
Subject: [PATCH 2/5] selftest: improve debugging in dns_hub.py
We only print debug messages when the response is delayed by more than 2
seconds.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
selftest/target/dns_hub.py | 43 +++++++++++++++++++++++++++++++-------
1 file changed, 36 insertions(+), 7 deletions(-)
diff --git a/selftest/target/dns_hub.py b/selftest/target/dns_hub.py
index e0fe68e985f6..2cfc8d076aa8 100755
--- a/selftest/target/dns_hub.py
+++ b/selftest/target/dns_hub.py
@@ -24,6 +24,7 @@ import threading
import sys
import select
import socket
+import time
from samba.dcerpc import dns
import samba.ndr as ndr
@@ -38,6 +39,16 @@ DNS_REQUEST_TIMEOUT = 10
class DnsHandler(sserver.BaseRequestHandler):
+ dns_qtype_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_QTYPE_'))
+ def dns_qtype_string(self, qtype):
+ "Return a readable qtype code"
+ return self.dns_qtype_strings[qtype]
+
+ dns_rcode_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_RCODE_'))
+ def dns_rcode_string(self, rcode):
+ "Return a readable error code"
+ return self.dns_rcode_strings[rcode]
+
def dns_transaction_udp(self, packet, host):
"send a DNS query and read the reply"
s = None
@@ -65,6 +76,13 @@ class DnsHandler(sserver.BaseRequestHandler):
return 'ignore'
if lname.endswith('dsfsdfs'):
return 'fail'
+ if lname.endswith("torture1", 0, len(lname)-2):
+ # CATCH TORTURE100, TORTURE101, ...
+ return 'torture'
+ if lname.endswith('_none_.example.com'):
+ return 'torture'
+ if lname.endswith('torturedom.samba.example.com'):
+ return 'torture'
if lname.endswith('adnonssdom.samba.example.com'):
return '127.0.0.17'
if lname.endswith('adnontlmdom.samba.example.com'):
@@ -92,6 +110,7 @@ class DnsHandler(sserver.BaseRequestHandler):
return None
def handle(self):
+ start = time.monotonic()
data, sock = self.request
query = ndr.ndr_unpack(dns.name_packet, data)
name = query.questions[0].name
@@ -102,13 +121,13 @@ class DnsHandler(sserver.BaseRequestHandler):
return
elif forwarder is 'fail':
pass
- elif forwarder is not None:
- response = self.dns_transaction_udp(query, forwarder)
- else:
+ elif forwarder in ['torture', None]:
response = query
response.operation |= dns.DNS_FLAG_REPLY
response.operation |= dns.DNS_FLAG_RECURSION_AVAIL
response.operation |= dns.DNS_RCODE_NXDOMAIN
+ else:
+ response = self.dns_transaction_udp(query, forwarder)
if response is None:
response = query
@@ -118,14 +137,24 @@ class DnsHandler(sserver.BaseRequestHandler):
send_packet = ndr.ndr_pack(response)
- print("dns_hub: sending %s to address %s for name %s\n" %
- (forwarder, self.client_address, name))
+ end = time.monotonic()
+ tdiff = end - start
+ errcode = response.operation & dns.DNS_RCODE
+ if tdiff > (DNS_REQUEST_TIMEOUT/5):
+ debug = True
+ else:
+ debug = False
+ if debug:
+ print("dns_hub: forwarder[%s] client[%s] name[%s][%s] %s response.operation[0x%x] tdiff[%s]\n" %
+ (forwarder, self.client_address, name,
+ self.dns_qtype_string(query.questions[0].question_type),
+ self.dns_rcode_string(errcode), response.operation, tdiff))
try:
sock.sendto(send_packet, self.client_address)
except socket.error as err:
- print("Error sending %s to address %s for name %s: %s\n" %
- (forwarder, self.client_address, name, err))
+ print("dns_hub: Error sending response to client[%s] for name[%s] tdiff[%s]: %s\n" %
+ (self.client_address, name, tdiff, err))
class server_thread(threading.Thread):
--
2.17.1
From 6c91520bcc3071758c1982873254b67caf6cd19d Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 8 Feb 2019 00:19:56 +0100
Subject: [PATCH 3/5] blackbox/dbcheck.sh: fix dbcheck_fix_one_way_links
cleanup
Commit 35bfc62a31c9ad73449594ddd48f76f50e0abade changed
dbcheck to not regard old one-way-links as errors.
At that time the relavant trigger changed from
fix_all_string_dn_component_mismatch to
fix_all_old_dn_string_component_mismatch.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
testprogs/blackbox/dbcheck.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testprogs/blackbox/dbcheck.sh b/testprogs/blackbox/dbcheck.sh
index 5691426d01b4..caf89ca402b7 100755
--- a/testprogs/blackbox/dbcheck.sh
+++ b/testprogs/blackbox/dbcheck.sh
@@ -19,7 +19,7 @@ dbcheck() {
# This list of attributes can be freely extended
dbcheck_fix_one_way_links() {
- $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_string_dn_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS
+ $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_old_dn_string_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS
}
# This list of attributes can be freely extended
--
2.17.1
From 673f221ad8444b30c1afb3d04980f2ebbd4bbf06 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 8 Feb 2019 10:57:13 +0100
Subject: [PATCH 4/5] dsdb/tests/vlv: use only one toplevel dn that is
correctly cleaned up
Before "OU=vlvtestou2,%s" % (self.base_dn) was left behind after the
test.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
source4/dsdb/tests/python/vlv.py | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/source4/dsdb/tests/python/vlv.py b/source4/dsdb/tests/python/vlv.py
index 8550a38e2870..29cc1718c78d 100644
--- a/source4/dsdb/tests/python/vlv.py
+++ b/source4/dsdb/tests/python/vlv.py
@@ -153,12 +153,16 @@ class TestsWithUserOU(samba.tests.TestCase):
session_info=system_session(lp), lp=lp)
self.base_dn = self.ldb.domain_dn()
- self.ou = "ou=vlv,%s" % self.base_dn
+ self.tree_dn = "ou=vlvtesttree,%s" % self.base_dn
+ self.ou = "ou=vlvou,%s" % self.tree_dn
if opts.delete_in_setup:
try:
- self.ldb.delete(self.ou, ['tree_delete:1'])
+ self.ldb.delete(self.tree_dn, ['tree_delete:1'])
except ldb.LdbError as e:
- print("tried deleting %s, got error %s" % (self.ou, e))
+ print("tried deleting %s, got error %s" % (self.tree_dn, e))
+ self.ldb.add({
+ "dn": self.tree_dn,
+ "objectclass": "organizationalUnit"})
self.ldb.add({
"dn": self.ou,
"objectclass": "organizationalUnit"})
@@ -191,7 +195,7 @@ class TestsWithUserOU(samba.tests.TestCase):
def tearDown(self):
super(TestsWithUserOU, self).tearDown()
if not opts.delete_in_setup:
- self.ldb.delete(self.ou, ['tree_delete:1'])
+ self.ldb.delete(self.tree_dn, ['tree_delete:1'])
class VLVTests(TestsWithUserOU):
@@ -1368,11 +1372,7 @@ class PagedResultsTests(TestsWithUserOU):
def test_paged_modify_object_scope(self):
expr = "(objectClass=*)"
- ou2 = "OU=vlvtestou2,%s" % (self.base_dn)
- try:
- self.ldb.delete(ou2, ['tree_delete:1'])
- except ldb.LdbError:
- pass
+ ou2 = "OU=vlvtestou2,%s" % (self.tree_dn)
self.ldb.add({"dn": ou2, "objectclass": "organizationalUnit"})
# Do a separate, full search to get all results
--
2.17.1
From 4e362dcb51929140c3b4551b76a9d4561d653cac Mon Sep 17 00:00:00 2001
From: Ralph Boehme <slow at samba.org>
Date: Wed, 23 Jan 2019 09:43:33 +0100
Subject: [PATCH 5/5] CI: split out "samba-ad-dc-ntvfs[-py2]" test targets
Many AD tests currently use the "samba" target. Split out a new target
"samba-ad-dc-ntvfs" and have all tests that use the "ad_dc_ntvfs" env
use the new target. This should greatly speed up the runtime for the "samba"
target and avoid swapping.
This reduces the total CI time by ~ 55%, I got an autobuild and a gitlab
pipeline finished in just ~ 100 mins!
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
Signed-off-by: Ralph Boehme <slow at samba.org>
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
.gitlab-ci.yml | 10 ++++++++++
script/autobuild.py | 26 ++++++++++++++++++++------
2 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 908c29ec9d95..0729599e1b87 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -118,3 +118,13 @@ build_samba_buildpy2_only:
script:
- python script/autobuild.py samba-buildpy2-only --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc_ntvfs:
+ <<: *shared_template
+ script:
+ # this one takes about 100 mins to finish
+ - script/autobuild.py samba-ad-dc-ntvfs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
+build_samba_ad_dc_ntvfs_py2:
+ <<: *shared_template
+ script:
+ - script/autobuild.py samba-ad-dc-ntvfs-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
diff --git a/script/autobuild.py b/script/autobuild.py
index 00f0d2202a3d..b1bd9d1686c0 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -49,6 +49,8 @@ builddirs = {
"samba-none-env": ".",
"samba-ad-dc": ".",
"samba-ad-dc-py2": ".",
+ "samba-ad-dc-ntvfs": ".",
+ "samba-ad-dc-ntvfs-py2": ".",
"samba-ad-dc-2": ".",
"samba-ad-dc-2-py2": ".",
"samba-ad-dc-backup": ".",
@@ -93,8 +95,10 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)'
- "samba": [("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ # We have 'test' before 'install' because, 'test' should work without 'install (runs all the other envs)'
+ "samba": [
+ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1 "
"TESTS='${PY3_ONLY}"
@@ -102,6 +106,7 @@ tasks = {
"--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
+ "--exclude-env=ad_dc_ntvfs "
"--exclude-env=ad_dc_no_nss "
"--exclude-env=fl2003dc "
"--exclude-env=fl2008r2dc "
@@ -123,7 +128,6 @@ tasks = {
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We split out this so the isolated nt4_dc tests do not wait for ad_dc or ad_dc_ntvfs tests (which are long)
"samba-nt4": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ads --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -134,7 +138,6 @@ tasks = {
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-fileserver": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ad-dc --without-ldap --without-ads --without-json --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -143,7 +146,6 @@ tasks = {
"--include-env=fileserver'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -157,7 +159,6 @@ tasks = {
"--include-env=ad_member_idmap_ad'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc-2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -171,6 +172,19 @@ tasks = {
"text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+ # We split out the ad_dc_ntvfs tests (which are long) so other test do not wait
+ # This is currently the longest task, so we don't randomly delay it.
+ "samba-ad-dc-ntvfs": [
+ ("random-sleep", "script/random-sleep.sh 1 1", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=ad_dc_ntvfs "
+ "'",
+ "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
# run the backup/restore testenvs separately as they're fairly standalone
# (and CI seems to max out at ~8 different DCs running at once)
"samba-ad-dc-backup": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
--
2.17.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190211/9baca110/signature.sig>
More information about the samba-technical
mailing list