[PATCH] Avoid inheriting *.stdout and *.stderr into every child process

[Andrew Bartlett]

Some of our processes are coming very close to the 512 process limit in
buildnice.  These log files from autobuild however don't need to be
open in every child.

Under CI here:
 https://gitlab.com/catalyst-samba/samba/pipelines/28743555

Please review so I can push with the py3 autobuild changes (or push if
you desire).

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

-------------- next part --------------
From 81da6fdf325257d151fdd69074ce83652123b878 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 27 Aug 2018 21:00:58 +1200
Subject: [PATCH 1/2] autobuild: use close_fds=True to avoid *.stderr and
 *.stdout inheriting into every process

Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
 script/autobuild.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/script/autobuild.py b/script/autobuild.py
index 9f20b65f692..6ad1941890b 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -407,7 +407,7 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
     if show:
         do_print("Running: '%s' in '%s'" % (cmd, dir))
     if output:
-        return Popen([cmd], shell=True, stdout=PIPE, cwd=dir).communicate()[0]
+        return Popen([cmd], shell=True, stdout=PIPE, cwd=dir, close_fds=True).communicate()[0]
     elif checkfail:
         return check_call(cmd, shell=True, cwd=dir)
     else:
@@ -473,6 +473,7 @@ class builder(object):
         cwd = os.getcwd()
         os.chdir("%s/%s" % (self.sdir, self.dir))
         self.proc = Popen(self.cmd, shell=True,
+                          close_fds=True,
                           stdout=self.stdout, stderr=self.stderr, stdin=self.stdin)
         os.chdir(cwd)
         self.next += 1
@@ -615,7 +616,7 @@ class buildlist(object):
         cwd = os.getcwd()
         cmd = "tail -f *.stdout *.stderr"
         os.chdir(gitroot)
-        self.tail_proc = Popen(cmd, shell=True)
+        self.tail_proc = Popen(cmd, shell=True, close_fds=True)
         os.chdir(cwd)
 
 
-- 
2.17.1


From 43ce52ea4d35f2fcab4e0b4addc2063de10a0747 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 27 Aug 2018 21:13:29 +1200
Subject: [PATCH 2/2] autobuild: Avoid subshell for tail -f invocation

This should mean one less process in the process tree, and less places to hold
FDs open.

Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
 script/autobuild.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/script/autobuild.py b/script/autobuild.py
index 6ad1941890b..fec64094c00 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -613,11 +613,11 @@ class buildlist(object):
             os.unlink(b.stderr_path)
 
     def start_tail(self):
-        cwd = os.getcwd()
-        cmd = "tail -f *.stdout *.stderr"
-        os.chdir(gitroot)
-        self.tail_proc = Popen(cmd, shell=True, close_fds=True)
-        os.chdir(cwd)
+        cmd = ["tail", "-f"]
+        for b in self.tlist:
+            cmd.append(b.stdout_path)
+            cmd.append(b.stderr_path)
+        self.tail_proc = Popen(cmd, close_fds=True)
 
 
 def cleanup():
-- 
2.17.1