pytalloc patches

Petr Viktorin pviktori at redhat.com
Mon May 11 06:56:15 MDT 2015


Hello,
After some off-list discussion with Thomas Nagy about modifications to
waf, I am submitting reworked buildsystem/pytalloc patches for another
round of review.
This version works with a copy of the build environment, so the changes
are less invasive.


With Thomas' permission, I'm forwarding the discussion:
Thomas Nagy wrote:
> 
> The latest changes provide a much better separation of concerns, and they are easier to review. They are mostly fine.
> 
> One suggestion for future changes:
> +        for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
> +            bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
> The best practice is to adjust environment settings during the configuration phase so that:
> * duplication of initialization code is prevented
> * accidental dependencies between build scripts are not created (ideally, build scripts could be read in any particular order)
> * analysis and debugging is simplified (tweak variables without any project reconfiguration)
> 
> When environment objects differ too much to be merged, it is possible to:
> * override the method ConfigurationContext.store() to perform last-minute adjustments
> * or to use a global bld.add_pre_fun(function) to perform a global initialization
> 
> Thomas
> 
> On Wed, 06 May 2015 19:23:41 +0200, Petr Viktorin wrote:
>> On 05/04/2015 08:57 PM, Thomas Nagy wrote:
>>> On Mon, 04 May 2015 17:02:03 +0200, Petr Viktorin wrote:
>>>
>>>> On 05/03/2015 09:19 PM, T N wrote:
>>>>> The patches above may work, but what would happen if a yet another
>>>>> python version became necessary?
>>>>
>>>> As in building for *three* different versions of Python? It would still
>>>> be possible, just not simultaneously. Build for two, then reconfigure
>>>> and build for the third.
>>>>
>>>> I agree that my solution is not general in this respect, but I don't
>>>> think it's a practical concern.
>>>>
>>>>> Since the build holds all the data in
>>>>> the conf.env/bld.env objects, it may be more simple to either:
>>>>> * Set another conf.env/bld.env before declaring tests/targets, without
>>>>> swapping variables manually and without additional variables. For
>>>>> example, to have all targets in a particular file use that particular
>>>>> interpreter, set bld.env=bld.all_envs['fancy_extra_python'] on top of
>>>>> the file and reset it with bld.env=bld.all_envs['default'] at the end.
>>>>
>>>> Well, Build.env is defined as:
>>>>
>>>> 	def get_env(self):
>>>> 		return self.env_of_name('default')
>>>> 	def set_env(self, name, val):
>>>> 		self.all_envs[name] = val
>>>>
>>>> 	env = property(get_env, set_env)
>>>>
>>>> So `bld.env=X` calls `set_env(bld, X)`, which fails with wrong number of
>>>> arguments. Even if it did work, bld.env is hard-wired to be
>>>> bld.env_of_name('default').
>>>
>>> And what would be the problem if it worked? Are the targets using that python version grouped together in the same wscript file? in this case - in pseudocode - it would suffice to use the other env object in the file:
>>>
>>> """
>>> bak = bld.env
>>> bld.all_envs['default'] = bld.all_envs['extrapython']
>>>
>>> bld.SAMBA_TARGET(...)
>>> bld.SAMBA_LIBRARY(...)
>>>
>>> bld.all_envs['default'] = bak
>>> """
>>>
>>> and the environment can be prepared during the configuration through:
>>> """
>>>         env = conf.all_envs['extrapython'] = conf.env.copy()
>>>         env.EXTRAPYTHON = '/foo'
>>> """
>>
>> A problem is that the copy is only linked to its parent during the
>> configure step. For the build step, it's deserialized from a file, so
>> any changes to the default env aren't reflected in the copy.
>> Samba records the dependencies in the env. So if I have:
>>
>>    bld.SAMBA_LIBRARY('talloc', ...)
>>
>>    bld.all_envs['default'] = bld.all_envs['extrapython']
>>
>>    bld.SAMBA_LIBRARY(..., public_deps='talloc')
>>
>> where "talloc" is not Python-specific (and thus shared between the two
>> versions of the bindings), the dependency is not found.
>>
>> So I'd need to copy a bunch of variables back to the extrapython env
>> when choosing it. There's a lot fewer of them than in the list in the
>> patches ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR' are the
>> ones I found), but they're not Python-specific, and would have to be
>> updated when something outside of the Python support adds one.
>>
>> That worries me a bit, but it's definitely a less invasive approach.
>>
>>> Or is it that the targets are spread over several files?
>>
>> Yes, eventually each standalone library will need its own bindings. But
>> I don't think that's a problem.
>>
>>>>> This may also provide a better separation of concern by avoiding the
>>>>> addition of new parameters to all the functions above.
>>>>> * Or to just update bld.env with the desired variables once. For example
>>>>> in init_extrapyext: self.env=self.env.copy;self.env.table.update(...) ).
>>>>> Here too a specific environment object can be prepared during the
>>>>> configuration.
>>>>
>>>> Same problem here; I can't modify self.env (i.e.
>>>> self.env_of_name('default')) without affecting all other uses of the env.
>>>
>>> The object "self.env" is a copy of bld.env, and it is definitely meant to be modified (use the methods append_value/append_unique). Each task generator is holding a copy (shallow copy), and each task object is also holding copy of the task generator env copy.
>>>
>>> Am I misunderstanding the intention, i.e. the code is actually merging object code of different python versions into the same libraries/programs?
[...]
>> I'm attaching a reworked set of patches. You can ignore the first one
>> for now, it's not related to the build system.
>> Does this look better to you?
>>
>> (I'm not yet convinced on the 5th patch, "Add a helper to iterate
>> through Python": it reduces boilerplate, but perhaps it's too magic to
>> have a for loop's iterator with side effects. But that's just a detail.)

-------------- next part --------------
From eb70ce83a9f8d4ce7e1b10cc649d412733c6704f Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Thu, 15 Jan 2015 14:07:09 +0100
Subject: [PATCH 1/7] pytalloc: Port to Python 3

- Use native string for repr

- Use rich comparison
  Removes the deprecated tp_compare in favor of tp_richcompare.
  Disparate types cannot be compared (except for == and !=),
  and True or False objects are returned explicitly.

- Use Py_TYPE instead of ob_type
  This changed to conform to C aliasing rules,
  see http://legacy.python.org/dev/peps/pep-3123/

- Don't provide CObject creation function
  A PyCapsule based replacement would be possible,
  but might not be necessary considering the function is
  not used much.

- Use new-style module initialization

Build changes:

- Use ABI flag in the lib name and pkg-config template

- Use the SAMBA_CHECK_PYTHON macro for finding Python

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 lib/talloc/pytalloc-util.pc.in |  2 +-
 lib/talloc/pytalloc.c          | 87 ++++++++++++++++++++++++++++++++++++++----
 lib/talloc/pytalloc.h          |  2 +
 lib/talloc/pytalloc_guide.txt  | 10 +++++
 lib/talloc/pytalloc_util.c     |  4 ++
 lib/talloc/test_pytalloc.c     | 41 +++++++++++++++++---
 lib/talloc/test_pytalloc.py    |  3 ++
 lib/talloc/wscript             | 10 ++---
 8 files changed, 140 insertions(+), 19 deletions(-)

diff --git a/lib/talloc/pytalloc-util.pc.in b/lib/talloc/pytalloc-util.pc.in
index b7426bb..b87c94e 100644
--- a/lib/talloc/pytalloc-util.pc.in
+++ b/lib/talloc/pytalloc-util.pc.in
@@ -6,6 +6,6 @@ includedir=@includedir@
 Name: pytalloc-util
 Description: Utility functions for using talloc objects with Python
 Version: @TALLOC_VERSION@
-Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util
+Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util at PYTHON_SO_ABI_FLAG@
 Cflags: -I${includedir}
 URL: http://talloc.samba.org/
diff --git a/lib/talloc/pytalloc.c b/lib/talloc/pytalloc.c
index ac4fe0f..3afae9c 100644
--- a/lib/talloc/pytalloc.c
+++ b/lib/talloc/pytalloc.c
@@ -21,7 +21,13 @@
 #include <talloc.h>
 #include <pytalloc.h>
 
-void inittalloc(void);
+static PyTypeObject TallocObject_Type;
+
+#if PY_MAJOR_VERSION >= 3
+#define PyStr_FromFormat PyUnicode_FromFormat
+#else
+#define PyStr_FromFormat PyString_FromFormat
+#endif
 
 /* print a talloc tree report for a talloc python object */
 static PyObject *pytalloc_report_full(PyObject *self, PyObject *args)
@@ -79,8 +85,8 @@ static PyObject *pytalloc_default_repr(PyObject *obj)
 	pytalloc_Object *talloc_obj = (pytalloc_Object *)obj;
 	PyTypeObject *type = (PyTypeObject*)PyObject_Type(obj);
 
-	return PyString_FromFormat("<%s talloc object at 0x%p>", 
-				   type->tp_name, talloc_obj->ptr);
+	return PyStr_FromFormat("<%s talloc object at 0x%p>",
+				type->tp_name, talloc_obj->ptr);
 }
 
 /**
@@ -97,6 +103,35 @@ static void pytalloc_dealloc(PyObject* self)
 /**
  * Default (but only slightly more useful than the default) implementation of cmp.
  */
+#if PY_MAJOR_VERSION >= 3
+static PyObject *pytalloc_default_richcmp(PyObject *obj1, PyObject *obj2, int op)
+{
+	void *ptr1;
+	void *ptr2;
+	if (Py_TYPE(obj1) == Py_TYPE(obj2)) {
+		/* When types match, compare pointers */
+		ptr1 = pytalloc_get_ptr(obj1);
+		ptr2 = pytalloc_get_ptr(obj2);
+	} else if (PyObject_TypeCheck(obj2, &TallocObject_Type)) {
+		/* Otherwise, compare types */
+		ptr1 = Py_TYPE(obj1);
+		ptr2 = Py_TYPE(obj2);
+	} else {
+		Py_INCREF(Py_NotImplemented);
+		return Py_NotImplemented;
+	}
+	switch (op) {
+		case Py_EQ: return PyBool_FromLong(ptr1 == ptr2);
+		case Py_NE: return PyBool_FromLong(ptr1 != ptr2);
+		case Py_LT: return PyBool_FromLong(ptr1 < ptr2);
+		case Py_GT: return PyBool_FromLong(ptr1 > ptr2);
+		case Py_LE: return PyBool_FromLong(ptr1 <= ptr2);
+		case Py_GE: return PyBool_FromLong(ptr1 >= ptr2);
+	}
+	Py_INCREF(Py_NotImplemented);
+	return Py_NotImplemented;
+}
+#else
 static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
 {
 	pytalloc_Object *obj1 = (pytalloc_Object *)_obj1,
@@ -106,6 +141,7 @@ static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
 
 	return ((char *)pytalloc_get_ptr(obj1) - (char *)pytalloc_get_ptr(obj2));
 }
+#endif
 
 static PyTypeObject TallocObject_Type = {
 	.tp_name = "talloc.Object",
@@ -114,21 +150,56 @@ static PyTypeObject TallocObject_Type = {
 	.tp_dealloc = (destructor)pytalloc_dealloc,
 	.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
 	.tp_repr = pytalloc_default_repr,
+#if PY_MAJOR_VERSION >= 3
+	.tp_richcompare = pytalloc_default_richcmp,
+#else
 	.tp_compare = pytalloc_default_cmp,
+#endif
 };
 
-void inittalloc(void)
+#define MODULE_DOC PyDoc_STR("Python wrapping of talloc-maintained objects.")
+
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef moduledef = {
+    PyModuleDef_HEAD_INIT,
+    .m_name = "talloc",
+    .m_doc = MODULE_DOC,
+    .m_size = -1,
+    .m_methods = talloc_methods,
+};
+#endif
+
+static PyObject *module_init(void);
+static PyObject *module_init(void)
 {
 	PyObject *m;
 
 	if (PyType_Ready(&TallocObject_Type) < 0)
-		return;
+		return NULL;
 
-	m = Py_InitModule3("talloc", talloc_methods,
-					   "Python wrapping of talloc-maintained objects.");
+#if PY_MAJOR_VERSION >= 3
+	m = PyModule_Create(&moduledef);
+#else
+	m = Py_InitModule3("talloc", talloc_methods, MODULE_DOC);
+#endif
 	if (m == NULL)
-		return;
+		return NULL;
 
 	Py_INCREF(&TallocObject_Type);
 	PyModule_AddObject(m, "Object", (PyObject *)&TallocObject_Type);
+	return m;
+}
+
+#if PY_MAJOR_VERSION >= 3
+PyMODINIT_FUNC PyInit_talloc(void);
+PyMODINIT_FUNC PyInit_talloc(void)
+{
+	return module_init();
+}
+#else
+void inittalloc(void);
+void inittalloc(void)
+{
+	module_init();
 }
+#endif
diff --git a/lib/talloc/pytalloc.h b/lib/talloc/pytalloc.h
index 5c3876e..608328e 100644
--- a/lib/talloc/pytalloc.h
+++ b/lib/talloc/pytalloc.h
@@ -52,6 +52,8 @@ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_ctx, void
 
 #define pytalloc_new(type, typeobj) pytalloc_steal(typeobj, talloc_zero(NULL, type))
 
+#if PY_MAJOR_VERSION < 3
 PyObject *pytalloc_CObject_FromTallocPtr(void *);
+#endif
 
 #endif /* _PYTALLOC_H_ */
diff --git a/lib/talloc/pytalloc_guide.txt b/lib/talloc/pytalloc_guide.txt
index 755a52b..36ae5ff 100644
--- a/lib/talloc/pytalloc_guide.txt
+++ b/lib/talloc/pytalloc_guide.txt
@@ -20,6 +20,14 @@ for objects that wrap talloc-maintained memory in C. It won't write your
 bindings for you but it will make it easier to write C bindings that involve
 talloc, and take away some of the boiler plate.
 
+Python 3
+--------
+
+pytalloc can be used with Python 3. Usage from Python extension remains
+the same, but for the C utilities, the library to link to is tagged with
+Python's PEP3149 ABI tag, for example "pytalloc.cpython34m".
+To make a build for Python 3, configure with PYTHON=/usr/bin/python3.
+.
 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 pytalloc_Object
 
@@ -126,6 +134,8 @@ use a generic VoidPtr Python type, which just provides an opaque object in
 Python. The caller is responsible for incrementing the talloc reference count before calling
 this function - it will dereference the talloc pointer when it is garbage collected.
 
+This function is only available on Python 2.
+
 Debug function for talloc in Python
 -----------------------------------
 
diff --git a/lib/talloc/pytalloc_util.c b/lib/talloc/pytalloc_util.c
index 89a093b..0af7c05 100644
--- a/lib/talloc/pytalloc_util.c
+++ b/lib/talloc/pytalloc_util.c
@@ -97,6 +97,8 @@ _PUBLIC_ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_
 	return (PyObject *)ret;
 }
 
+#if PY_MAJOR_VERSION < 3
+
 static void py_cobject_talloc_free(void *ptr)
 {
 	talloc_free(ptr);
@@ -110,6 +112,8 @@ _PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
 	return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
 }
 
+#endif
+
 _PUBLIC_ int pytalloc_Check(PyObject *obj)
 {
 	PyTypeObject *tp = pytalloc_GetObjectType();
diff --git a/lib/talloc/test_pytalloc.c b/lib/talloc/test_pytalloc.c
index 2eaa7c3..f66b4e5 100644
--- a/lib/talloc/test_pytalloc.c
+++ b/lib/talloc/test_pytalloc.c
@@ -104,25 +104,56 @@ static PyTypeObject DObject_Type = {
 	.tp_doc = "test talloc object that calls a function when underlying data is freed\n",
 };
 
-#define MODULE_DOC "Test utility module for pytalloc"
+#define MODULE_DOC PyDoc_STR("Test utility module for pytalloc")
+
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef moduledef = {
+    PyModuleDef_HEAD_INIT,
+    .m_name = "_test_pytalloc",
+    .m_doc = PyDoc_STR("Test utility module for pytalloc"),
+    .m_size = -1,
+    .m_methods = test_talloc_methods,
+};
+#endif
 
-void init_test_pytalloc(void);
-void init_test_pytalloc(void)
+static PyObject *module_init(void);
+static PyObject *module_init(void)
 {
 	PyObject *m;
 
 	DObject_Type.tp_base = pytalloc_GetObjectType();
 	if (PyType_Ready(&DObject_Type) < 0) {
-		return;
+		return NULL;
 	}
 
+#if PY_MAJOR_VERSION >= 3
+	m = PyModule_Create(&moduledef);
+#else
 	m = Py_InitModule3("_test_pytalloc", test_talloc_methods, MODULE_DOC);
+#endif
 
 	if (m == NULL) {
-		return;
+		return NULL;
 	}
 
 	Py_INCREF(&DObject_Type);
 	Py_INCREF(DObject_Type.tp_base);
 	PyModule_AddObject(m, "DObject", (PyObject *)&DObject_Type);
+
+	return m;
+}
+
+
+#if PY_MAJOR_VERSION >= 3
+PyMODINIT_FUNC PyInit__test_pytalloc(void);
+PyMODINIT_FUNC PyInit__test_pytalloc(void)
+{
+	return module_init();
+}
+#else
+void init_test_pytalloc(void);
+void init_test_pytalloc(void)
+{
+	module_init();
 }
+#endif
diff --git a/lib/talloc/test_pytalloc.py b/lib/talloc/test_pytalloc.py
index 961bfcb..a613373 100644
--- a/lib/talloc/test_pytalloc.py
+++ b/lib/talloc/test_pytalloc.py
@@ -78,6 +78,9 @@ class TallocComparisonTests(unittest.TestCase):
 
     def test_compare_different_types(self):
         # object comparison falls back to comparing types
+        if sys.version_info >= (3, 0):
+            # In Python 3, types are unorderable -- nothing to test
+            return
         if talloc.Object < _test_pytalloc.DObject:
             obj1 = _test_pytalloc.new()
             obj2 = _test_pytalloc.DObject(dummy_func)
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 1367988..3bc932e 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -60,9 +60,7 @@ def configure(conf):
 
     if not conf.env.disable_python:
         # also disable if we don't have the python libs installed
-        conf.find_program('python', var='PYTHON')
-        conf.check_tool('python')
-        conf.check_python_version((2,4,2))
+        conf.SAMBA_CHECK_PYTHON(mandatory=False, version=(2,4,2))
         conf.SAMBA_CHECK_PYTHON_HEADERS(mandatory=False)
         if not conf.env.HAVE_PYTHON_H:
             Logs.warn('Disabling pytalloc-util as python devel libs not found')
@@ -118,7 +116,9 @@ def build(bld):
                           manpages='man/talloc.3')
 
     if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
-        bld.SAMBA_LIBRARY('pytalloc-util',
+        name = bld.pyembed_libname('pytalloc-util')
+
+        bld.SAMBA_LIBRARY(name,
             source='pytalloc_util.c',
             public_deps='talloc',
             pyembed=True,
@@ -132,7 +132,7 @@ def build(bld):
             )
         bld.SAMBA_PYTHON('pytalloc',
                          'pytalloc.c',
-                         deps='talloc pytalloc-util',
+                         deps='talloc ' + name,
                          enabled=True,
                          realname='talloc.so')
 
-- 
2.1.0


From c1c0d13e119e4323f6cd999a45af4db03dde1ce1 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 12:45:42 +0200
Subject: [PATCH 2/7] buildtools: Expose the Python 3 ABI tag

Expose the tag in the env to allow using it in pkg-config files

Add a "pyembed_libname" function to correctly form library names.
(This can't be done automatically in SAMBA_LIBRARY because the name
could be used as a dependency for another library)

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 buildtools/wafsamba/samba_python.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index a371b43..b90655d 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -23,6 +23,12 @@ def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
     else:
         conf.msg("python headers", "using cache")
 
+    if conf.env['PYTHON_VERSION'] > '3':
+        abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
+        conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
+    else:
+        conf.env['PYTHON_SO_ABI_FLAG'] = ''
+
 
 def SAMBA_PYTHON(bld, name,
                  source='',
@@ -69,3 +75,9 @@ def SAMBA_PYTHON(bld, name,
                       enabled=enabled)
 
 Build.BuildContext.SAMBA_PYTHON = SAMBA_PYTHON
+
+
+def pyembed_libname(bld, name, extrapython=False):
+    return name + bld.env['PYTHON_SO_ABI_FLAG']
+
+Build.BuildContext.pyembed_libname = pyembed_libname
-- 
2.1.0


From cc09d48a3a234e23f114654bb6ed10d90337964a Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Thu, 15 Jan 2015 14:22:22 +0100
Subject: [PATCH 3/7] buildtools: Add --extra-python configure option

This allows building Python support for two different Python versions
at the same time.

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 buildtools/wafsamba/samba_install.py |  8 +++++++-
 buildtools/wafsamba/samba_python.py  | 32 +++++++++++++++++++++++++++++++-
 buildtools/wafsamba/wafsamba.py      |  4 ++--
 buildtools/wafsamba/wscript          |  8 ++++++++
 4 files changed, 48 insertions(+), 4 deletions(-)

diff --git a/buildtools/wafsamba/samba_install.py b/buildtools/wafsamba/samba_install.py
index aa7f143..af8d2ad 100644
--- a/buildtools/wafsamba/samba_install.py
+++ b/buildtools/wafsamba/samba_install.py
@@ -59,6 +59,10 @@ def install_library(self):
 
     bld = self.bld
 
+    default_env = bld.all_envs['default']
+    if self.env['IS_EXTRA_PYTHON']:
+        bld.all_envs['default'] = bld.all_envs['extrapython']
+
     install_ldflags = install_rpath(self)
     build_ldflags   = build_rpath(bld)
 
@@ -83,7 +87,7 @@ def install_library(self):
         # install link. That stops us from overwriting the existing build
         # target, which has different ldflags
         self.done_install_library = True
-        t = self.clone('default')
+        t = self.clone(self.env)
         t.posted = False
         t.target += '.inst'
         self.env.RPATH = build_ldflags
@@ -144,6 +148,8 @@ def install_library(self):
     if dev_link:
         bld.symlink_as(os.path.join(install_path, dev_link), os.path.basename(install_name))
 
+    bld.all_envs['default'] = default_env
+
 
 @feature('cshlib')
 @after('apply_implib')
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index b90655d..d12fe95 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -9,20 +9,50 @@ from Configure import conf
 @conf
 def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
     # enable tool to build python extensions
+    if conf.env['EXTRA_PYTHON']:
+        conf.all_envs['extrapython'] = conf.env.copy()
+        conf.setenv('extrapython')
+        conf.env['PYTHON'] = conf.env['EXTRA_PYTHON']
+        conf.env['IS_EXTRA_PYTHON'] = 'yes'
+        conf.find_program('python', var='PYTHON', mandatory=True)
+        conf.check_tool('python')
+        try:
+            conf.check_python_version((3, 3, 0))
+        except Exception:
+            warn('extra-python needs to be Python 3.3 or later')
+            raise
+        conf.setenv('default')
+
     conf.find_program('python', var='PYTHON', mandatory=mandatory)
     conf.check_tool('python')
     path_python = conf.find_program('python')
     conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
     conf.check_python_version(version)
 
+
 @conf
 def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
     if conf.env["python_headers_checked"] == []:
-        conf.check_python_headers(mandatory)
+        if conf.env['EXTRA_PYTHON']:
+            conf.setenv('extrapython')
+            _check_python_headers(conf, mandatory=True)
+            conf.setenv('default')
+
+        _check_python_headers(conf, mandatory)
         conf.env["python_headers_checked"] = "yes"
+
+        if conf.env['EXTRA_PYTHON']:
+            extraversion = conf.all_envs['extrapython']['PYTHON_VERSION']
+            if extraversion == conf.env['PYTHON_VERSION']:
+                raise Utils.WafError("extrapython %s is same as main python %s" % (
+                    extraversion, conf.env['PYTHON_VERSION']))
     else:
         conf.msg("python headers", "using cache")
 
+
+def _check_python_headers(conf, mandatory):
+    conf.check_python_headers(mandatory=mandatory)
+
     if conf.env['PYTHON_VERSION'] > '3':
         abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
         conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
index d7e482c..12bf231 100644
--- a/buildtools/wafsamba/wafsamba.py
+++ b/buildtools/wafsamba/wafsamba.py
@@ -217,10 +217,10 @@ def SAMBA_LIBRARY(bld, libname, source,
         if vnum is None and soname is None:
             raise Utils.WafError("public library '%s' must have a vnum" %
                     libname)
-        if pc_files is None:
+        if pc_files is None and not bld.env['IS_EXTRA_PYTHON']:
             raise Utils.WafError("public library '%s' must have pkg-config file" %
                        libname)
-        if public_headers is None:
+        if public_headers is None and not bld.env['IS_EXTRA_PYTHON']:
             raise Utils.WafError("public library '%s' must have header files" %
                        libname)
 
diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript
index 694147e..d6bb688 100755
--- a/buildtools/wafsamba/wscript
+++ b/buildtools/wafsamba/wscript
@@ -195,6 +195,12 @@ def set_options(opt):
                    help='tag release in git at the same time',
                    type='string', action='store', dest='TAG_RELEASE')
 
+    opt.add_option('--extra-python', type=str,
+                    help=("build selected libraries for the specified "
+                          "additional version of Python "
+                          "(example: --extra-python=/usr/bin/python3)"),
+                    metavar="PYTHON", dest='EXTRA_PYTHON', default=None)
+
 
 @wafsamba.runonce
 def configure(conf):
@@ -266,6 +272,8 @@ def configure(conf):
     conf.env.AUTOCONF_HOST  = Options.options.AUTOCONF_HOST
     conf.env.AUTOCONF_PROGRAM_PREFIX = Options.options.AUTOCONF_PROGRAM_PREFIX
 
+    conf.env.EXTRA_PYTHON = Options.options.EXTRA_PYTHON
+
     if (conf.env.AUTOCONF_HOST and
         conf.env.AUTOCONF_BUILD and
         conf.env.AUTOCONF_BUILD != conf.env.AUTOCONF_HOST):
-- 
2.1.0


From 078383449579ca658312f6a69b15baf6135f33a5 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 18:17:06 +0200
Subject: [PATCH 4/7] pytalloc: Build for two Python versions at once

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 lib/talloc/wscript | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 3bc932e..0b62d40 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -143,6 +143,43 @@ def build(bld):
                          realname='_test_pytalloc.so',
                          install=False)
 
+    if bld.env['EXTRA_PYTHON']:
+        for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
+            bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
+        bak = bld.all_envs['default']
+        bld.all_envs['default'] = bld.all_envs['extrapython']
+
+        name = bld.pyembed_libname('pytalloc-util')
+
+        bld.SAMBA_LIBRARY(name,
+            source='pytalloc_util.c',
+            public_deps='talloc',
+            pyembed=True,
+            vnum=VERSION,
+            hide_symbols=True,
+            abi_directory='ABI',
+            abi_match='pytalloc_*',
+            private_library=private_library,
+            #public_headers='pytalloc.h',
+            #pc_files='pytalloc-util.pc'
+            )
+
+        bld.SAMBA_PYTHON('extra-pytalloc',
+                         'pytalloc.c',
+                         deps='talloc ' + name,
+                         enabled=True,
+                         realname='talloc.so')
+
+        bld.SAMBA_PYTHON('extra-test_pytalloc',
+                         'test_pytalloc.c',
+                         deps='pytalloc',
+                         enabled=True,
+                         realname='_test_pytalloc.so',
+                         install=False)
+
+        bld.all_envs['default'] = bak
+
+
 def test(ctx):
     '''run talloc testsuite'''
     import Utils, samba_utils
-- 
2.1.0


From b52eec2f7446888ed5812f7c0b30612f06adfca0 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 17:50:57 +0200
Subject: [PATCH 5/7] buildtools: Add a helper to iterate through Python
 environments

This prevents code duplication to ensure the "extrapython" build
is the same as the normal one.

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 buildtools/wafsamba/samba_python.py | 25 +++++++++++
 buildtools/wafsamba/wafsamba.py     |  3 ++
 lib/talloc/wscript                  | 89 +++++++++++--------------------------
 3 files changed, 55 insertions(+), 62 deletions(-)

diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index d12fe95..a0e0ffa 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -74,6 +74,9 @@ def SAMBA_PYTHON(bld, name,
                  enabled=True):
     '''build a python extension for Samba'''
 
+    if bld.env['IS_EXTRA_PYTHON']:
+        name = 'extra-' + name
+
     # when we support static python modules we'll need to gather
     # the list from all the SAMBA_PYTHON() targets
     if init_function_sentinel is not None:
@@ -111,3 +114,25 @@ def pyembed_libname(bld, name, extrapython=False):
     return name + bld.env['PYTHON_SO_ABI_FLAG']
 
 Build.BuildContext.pyembed_libname = pyembed_libname
+
+
+def gen_python_environments(bld, extra_env_vars=()):
+    """Generate all Python environments
+
+    To be used in a for loop. Normally, the loop body will be executed once.
+
+    When --extra-python is used, the body will additionaly be executed
+    with the extra-python environment active.
+    """
+    yield
+
+    if bld.env['EXTRA_PYTHON']:
+        copied = ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE') + tuple(extra_env_vars)
+        for name in copied:
+            bld.all_envs['extrapython'][name] = bld.all_envs['default'][name]
+        default_env = bld.all_envs['default']
+        bld.all_envs['default'] = bld.all_envs['extrapython']
+        yield
+        bld.all_envs['default'] = default_env
+
+Build.BuildContext.gen_python_environments = gen_python_environments
diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
index 12bf231..64382da 100644
--- a/buildtools/wafsamba/wafsamba.py
+++ b/buildtools/wafsamba/wafsamba.py
@@ -143,6 +143,9 @@ def SAMBA_LIBRARY(bld, libname, source,
                   enabled=True):
     '''define a Samba library'''
 
+    if pyembed and bld.env['IS_EXTRA_PYTHON']:
+        public_headers = pc_files = None
+
     if LIB_MUST_BE_PRIVATE(bld, libname):
         private_library=True
 
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 0b62d40..c520294 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -116,68 +116,33 @@ def build(bld):
                           manpages='man/talloc.3')
 
     if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
-        name = bld.pyembed_libname('pytalloc-util')
-
-        bld.SAMBA_LIBRARY(name,
-            source='pytalloc_util.c',
-            public_deps='talloc',
-            pyembed=True,
-            vnum=VERSION,
-            hide_symbols=True,
-            abi_directory='ABI',
-            abi_match='pytalloc_*',
-            private_library=private_library,
-            public_headers='pytalloc.h',
-            pc_files='pytalloc-util.pc'
-            )
-        bld.SAMBA_PYTHON('pytalloc',
-                         'pytalloc.c',
-                         deps='talloc ' + name,
-                         enabled=True,
-                         realname='talloc.so')
-
-        bld.SAMBA_PYTHON('test_pytalloc',
-                         'test_pytalloc.c',
-                         deps='pytalloc',
-                         enabled=True,
-                         realname='_test_pytalloc.so',
-                         install=False)
-
-    if bld.env['EXTRA_PYTHON']:
-        for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
-            bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
-        bak = bld.all_envs['default']
-        bld.all_envs['default'] = bld.all_envs['extrapython']
-
-        name = bld.pyembed_libname('pytalloc-util')
-
-        bld.SAMBA_LIBRARY(name,
-            source='pytalloc_util.c',
-            public_deps='talloc',
-            pyembed=True,
-            vnum=VERSION,
-            hide_symbols=True,
-            abi_directory='ABI',
-            abi_match='pytalloc_*',
-            private_library=private_library,
-            #public_headers='pytalloc.h',
-            #pc_files='pytalloc-util.pc'
-            )
-
-        bld.SAMBA_PYTHON('extra-pytalloc',
-                         'pytalloc.c',
-                         deps='talloc ' + name,
-                         enabled=True,
-                         realname='talloc.so')
-
-        bld.SAMBA_PYTHON('extra-test_pytalloc',
-                         'test_pytalloc.c',
-                         deps='pytalloc',
-                         enabled=True,
-                         realname='_test_pytalloc.so',
-                         install=False)
-
-        bld.all_envs['default'] = bak
+        for env in bld.gen_python_environments(['PKGCONFIGDIR']):
+            name = bld.pyembed_libname('pytalloc-util')
+
+            bld.SAMBA_LIBRARY(name,
+                source='pytalloc_util.c',
+                public_deps='talloc',
+                pyembed=True,
+                vnum=VERSION,
+                hide_symbols=True,
+                abi_directory='ABI',
+                abi_match='pytalloc_*',
+                private_library=private_library,
+                public_headers='pytalloc.h',
+                pc_files='pytalloc-util.pc'
+                )
+            bld.SAMBA_PYTHON('pytalloc',
+                            'pytalloc.c',
+                            deps='talloc ' + name,
+                            enabled=True,
+                            realname='talloc.so')
+
+            bld.SAMBA_PYTHON('test_pytalloc',
+                            'test_pytalloc.c',
+                            deps='pytalloc',
+                            enabled=True,
+                            realname='_test_pytalloc.so',
+                            install=False)
 
 
 def test(ctx):
-- 
2.1.0


From c5fb718a8aabaf8434ad5a0d48436affb0d2d63a Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Tue, 10 Mar 2015 18:19:14 +0100
Subject: [PATCH 6/7] buildtools: Add a helper for running Python tests

Add the function samba_utils.RUN_PYTHON_TESTS for running a Python
test. When building for multiple Python versions, all are tested.

Also, add the list of configured Python interpreters to build config.

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 buildtools/wafsamba/samba_python.py |  6 ++++++
 buildtools/wafsamba/samba_utils.py  | 16 ++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index a0e0ffa..8b20066 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -9,6 +9,8 @@ from Configure import conf
 @conf
 def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
     # enable tool to build python extensions
+    interpreters = []
+
     if conf.env['EXTRA_PYTHON']:
         conf.all_envs['extrapython'] = conf.env.copy()
         conf.setenv('extrapython')
@@ -21,6 +23,7 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
         except Exception:
             warn('extra-python needs to be Python 3.3 or later')
             raise
+        interpreters.append(conf.env['PYTHON'])
         conf.setenv('default')
 
     conf.find_program('python', var='PYTHON', mandatory=mandatory)
@@ -29,6 +32,9 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
     conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
     conf.check_python_version(version)
 
+    interpreters.append(conf.env['PYTHON'])
+    conf.env.python_interpreters = interpreters
+
 
 @conf
 def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
diff --git a/buildtools/wafsamba/samba_utils.py b/buildtools/wafsamba/samba_utils.py
index e8bc0f3..540fe44 100644
--- a/buildtools/wafsamba/samba_utils.py
+++ b/buildtools/wafsamba/samba_utils.py
@@ -386,6 +386,22 @@ def RUN_COMMAND(cmd,
     return -1
 
 
+def RUN_PYTHON_TESTS(testfiles, pythonpath=None):
+    env = LOAD_ENVIRONMENT()
+    if pythonpath is None:
+        pythonpath = os.path.join(Utils.g_module.blddir, 'python')
+    result = 0
+    for interp in env.python_interpreters:
+        for testfile in testfiles:
+            cmd = "PYTHONPATH=%s %s %s" % (pythonpath, interp, testfile)
+            print('Running Python test with %s: %s' % (interp, testfile))
+            ret = RUN_COMMAND(cmd)
+            if ret:
+                print('Python test failed: %s' % cmd)
+                result = ret
+    return result
+
+
 # make sure we have md5. some systems don't have it
 try:
     from hashlib import md5
-- 
2.1.0


From d22b7758d07180e9755a07948751bf4c216c29ff Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 18:05:18 +0200
Subject: [PATCH 7/7] pytalloc: Test for all Python versions

Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
 lib/talloc/wscript | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index c520294..8e61516 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -148,18 +148,10 @@ def build(bld):
 def test(ctx):
     '''run talloc testsuite'''
     import Utils, samba_utils
-    env = samba_utils.LOAD_ENVIRONMENT()
     cmd = os.path.join(Utils.g_module.blddir, 'talloc_testsuite')
     ret = samba_utils.RUN_COMMAND(cmd)
     print("testsuite returned %d" % ret)
-    if 'USING_SYSTEM_PYTALLOC_UTIL' not in env.defines and not env.disable_python:
-        cmd = "PYTHONPATH=%s %s test_pytalloc.py" % (
-            os.path.join(Utils.g_module.blddir, 'python'),
-            env['PYTHON'],
-        )
-        pyret = samba_utils.RUN_COMMAND(cmd)
-    else:
-        pyret = 0
+    pyret = samba_utils.RUN_PYTHON_TESTS(['test_pytalloc.py'])
     print("python testsuite returned %d" % pyret)
     sys.exit(ret or pyret)
 
-- 
2.1.0


More information about the samba-technical mailing list