pytalloc patches
Petr Viktorin
pviktori at redhat.com
Mon May 11 06:56:15 MDT 2015
Hello,
After some off-list discussion with Thomas Nagy about modifications to
waf, I am submitting reworked buildsystem/pytalloc patches for another
round of review.
This version works with a copy of the build environment, so the changes
are less invasive.
With Thomas' permission, I'm forwarding the discussion:
Thomas Nagy wrote:
>
> The latest changes provide a much better separation of concerns, and they are easier to review. They are mostly fine.
>
> One suggestion for future changes:
> + for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
> + bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
> The best practice is to adjust environment settings during the configuration phase so that:
> * duplication of initialization code is prevented
> * accidental dependencies between build scripts are not created (ideally, build scripts could be read in any particular order)
> * analysis and debugging is simplified (tweak variables without any project reconfiguration)
>
> When environment objects differ too much to be merged, it is possible to:
> * override the method ConfigurationContext.store() to perform last-minute adjustments
> * or to use a global bld.add_pre_fun(function) to perform a global initialization
>
> Thomas
>
> On Wed, 06 May 2015 19:23:41 +0200, Petr Viktorin wrote:
>> On 05/04/2015 08:57 PM, Thomas Nagy wrote:
>>> On Mon, 04 May 2015 17:02:03 +0200, Petr Viktorin wrote:
>>>
>>>> On 05/03/2015 09:19 PM, T N wrote:
>>>>> The patches above may work, but what would happen if a yet another
>>>>> python version became necessary?
>>>>
>>>> As in building for *three* different versions of Python? It would still
>>>> be possible, just not simultaneously. Build for two, then reconfigure
>>>> and build for the third.
>>>>
>>>> I agree that my solution is not general in this respect, but I don't
>>>> think it's a practical concern.
>>>>
>>>>> Since the build holds all the data in
>>>>> the conf.env/bld.env objects, it may be more simple to either:
>>>>> * Set another conf.env/bld.env before declaring tests/targets, without
>>>>> swapping variables manually and without additional variables. For
>>>>> example, to have all targets in a particular file use that particular
>>>>> interpreter, set bld.env=bld.all_envs['fancy_extra_python'] on top of
>>>>> the file and reset it with bld.env=bld.all_envs['default'] at the end.
>>>>
>>>> Well, Build.env is defined as:
>>>>
>>>> def get_env(self):
>>>> return self.env_of_name('default')
>>>> def set_env(self, name, val):
>>>> self.all_envs[name] = val
>>>>
>>>> env = property(get_env, set_env)
>>>>
>>>> So `bld.env=X` calls `set_env(bld, X)`, which fails with wrong number of
>>>> arguments. Even if it did work, bld.env is hard-wired to be
>>>> bld.env_of_name('default').
>>>
>>> And what would be the problem if it worked? Are the targets using that python version grouped together in the same wscript file? in this case - in pseudocode - it would suffice to use the other env object in the file:
>>>
>>> """
>>> bak = bld.env
>>> bld.all_envs['default'] = bld.all_envs['extrapython']
>>>
>>> bld.SAMBA_TARGET(...)
>>> bld.SAMBA_LIBRARY(...)
>>>
>>> bld.all_envs['default'] = bak
>>> """
>>>
>>> and the environment can be prepared during the configuration through:
>>> """
>>> env = conf.all_envs['extrapython'] = conf.env.copy()
>>> env.EXTRAPYTHON = '/foo'
>>> """
>>
>> A problem is that the copy is only linked to its parent during the
>> configure step. For the build step, it's deserialized from a file, so
>> any changes to the default env aren't reflected in the copy.
>> Samba records the dependencies in the env. So if I have:
>>
>> bld.SAMBA_LIBRARY('talloc', ...)
>>
>> bld.all_envs['default'] = bld.all_envs['extrapython']
>>
>> bld.SAMBA_LIBRARY(..., public_deps='talloc')
>>
>> where "talloc" is not Python-specific (and thus shared between the two
>> versions of the bindings), the dependency is not found.
>>
>> So I'd need to copy a bunch of variables back to the extrapython env
>> when choosing it. There's a lot fewer of them than in the list in the
>> patches ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR' are the
>> ones I found), but they're not Python-specific, and would have to be
>> updated when something outside of the Python support adds one.
>>
>> That worries me a bit, but it's definitely a less invasive approach.
>>
>>> Or is it that the targets are spread over several files?
>>
>> Yes, eventually each standalone library will need its own bindings. But
>> I don't think that's a problem.
>>
>>>>> This may also provide a better separation of concern by avoiding the
>>>>> addition of new parameters to all the functions above.
>>>>> * Or to just update bld.env with the desired variables once. For example
>>>>> in init_extrapyext: self.env=self.env.copy;self.env.table.update(...) ).
>>>>> Here too a specific environment object can be prepared during the
>>>>> configuration.
>>>>
>>>> Same problem here; I can't modify self.env (i.e.
>>>> self.env_of_name('default')) without affecting all other uses of the env.
>>>
>>> The object "self.env" is a copy of bld.env, and it is definitely meant to be modified (use the methods append_value/append_unique). Each task generator is holding a copy (shallow copy), and each task object is also holding copy of the task generator env copy.
>>>
>>> Am I misunderstanding the intention, i.e. the code is actually merging object code of different python versions into the same libraries/programs?
[...]
>> I'm attaching a reworked set of patches. You can ignore the first one
>> for now, it's not related to the build system.
>> Does this look better to you?
>>
>> (I'm not yet convinced on the 5th patch, "Add a helper to iterate
>> through Python": it reduces boilerplate, but perhaps it's too magic to
>> have a for loop's iterator with side effects. But that's just a detail.)
-------------- next part --------------
From eb70ce83a9f8d4ce7e1b10cc649d412733c6704f Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Thu, 15 Jan 2015 14:07:09 +0100
Subject: [PATCH 1/7] pytalloc: Port to Python 3
- Use native string for repr
- Use rich comparison
Removes the deprecated tp_compare in favor of tp_richcompare.
Disparate types cannot be compared (except for == and !=),
and True or False objects are returned explicitly.
- Use Py_TYPE instead of ob_type
This changed to conform to C aliasing rules,
see http://legacy.python.org/dev/peps/pep-3123/
- Don't provide CObject creation function
A PyCapsule based replacement would be possible,
but might not be necessary considering the function is
not used much.
- Use new-style module initialization
Build changes:
- Use ABI flag in the lib name and pkg-config template
- Use the SAMBA_CHECK_PYTHON macro for finding Python
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
lib/talloc/pytalloc-util.pc.in | 2 +-
lib/talloc/pytalloc.c | 87 ++++++++++++++++++++++++++++++++++++++----
lib/talloc/pytalloc.h | 2 +
lib/talloc/pytalloc_guide.txt | 10 +++++
lib/talloc/pytalloc_util.c | 4 ++
lib/talloc/test_pytalloc.c | 41 +++++++++++++++++---
lib/talloc/test_pytalloc.py | 3 ++
lib/talloc/wscript | 10 ++---
8 files changed, 140 insertions(+), 19 deletions(-)
diff --git a/lib/talloc/pytalloc-util.pc.in b/lib/talloc/pytalloc-util.pc.in
index b7426bb..b87c94e 100644
--- a/lib/talloc/pytalloc-util.pc.in
+++ b/lib/talloc/pytalloc-util.pc.in
@@ -6,6 +6,6 @@ includedir=@includedir@
Name: pytalloc-util
Description: Utility functions for using talloc objects with Python
Version: @TALLOC_VERSION@
-Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util
+Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util at PYTHON_SO_ABI_FLAG@
Cflags: -I${includedir}
URL: http://talloc.samba.org/
diff --git a/lib/talloc/pytalloc.c b/lib/talloc/pytalloc.c
index ac4fe0f..3afae9c 100644
--- a/lib/talloc/pytalloc.c
+++ b/lib/talloc/pytalloc.c
@@ -21,7 +21,13 @@
#include <talloc.h>
#include <pytalloc.h>
-void inittalloc(void);
+static PyTypeObject TallocObject_Type;
+
+#if PY_MAJOR_VERSION >= 3
+#define PyStr_FromFormat PyUnicode_FromFormat
+#else
+#define PyStr_FromFormat PyString_FromFormat
+#endif
/* print a talloc tree report for a talloc python object */
static PyObject *pytalloc_report_full(PyObject *self, PyObject *args)
@@ -79,8 +85,8 @@ static PyObject *pytalloc_default_repr(PyObject *obj)
pytalloc_Object *talloc_obj = (pytalloc_Object *)obj;
PyTypeObject *type = (PyTypeObject*)PyObject_Type(obj);
- return PyString_FromFormat("<%s talloc object at 0x%p>",
- type->tp_name, talloc_obj->ptr);
+ return PyStr_FromFormat("<%s talloc object at 0x%p>",
+ type->tp_name, talloc_obj->ptr);
}
/**
@@ -97,6 +103,35 @@ static void pytalloc_dealloc(PyObject* self)
/**
* Default (but only slightly more useful than the default) implementation of cmp.
*/
+#if PY_MAJOR_VERSION >= 3
+static PyObject *pytalloc_default_richcmp(PyObject *obj1, PyObject *obj2, int op)
+{
+ void *ptr1;
+ void *ptr2;
+ if (Py_TYPE(obj1) == Py_TYPE(obj2)) {
+ /* When types match, compare pointers */
+ ptr1 = pytalloc_get_ptr(obj1);
+ ptr2 = pytalloc_get_ptr(obj2);
+ } else if (PyObject_TypeCheck(obj2, &TallocObject_Type)) {
+ /* Otherwise, compare types */
+ ptr1 = Py_TYPE(obj1);
+ ptr2 = Py_TYPE(obj2);
+ } else {
+ Py_INCREF(Py_NotImplemented);
+ return Py_NotImplemented;
+ }
+ switch (op) {
+ case Py_EQ: return PyBool_FromLong(ptr1 == ptr2);
+ case Py_NE: return PyBool_FromLong(ptr1 != ptr2);
+ case Py_LT: return PyBool_FromLong(ptr1 < ptr2);
+ case Py_GT: return PyBool_FromLong(ptr1 > ptr2);
+ case Py_LE: return PyBool_FromLong(ptr1 <= ptr2);
+ case Py_GE: return PyBool_FromLong(ptr1 >= ptr2);
+ }
+ Py_INCREF(Py_NotImplemented);
+ return Py_NotImplemented;
+}
+#else
static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
{
pytalloc_Object *obj1 = (pytalloc_Object *)_obj1,
@@ -106,6 +141,7 @@ static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
return ((char *)pytalloc_get_ptr(obj1) - (char *)pytalloc_get_ptr(obj2));
}
+#endif
static PyTypeObject TallocObject_Type = {
.tp_name = "talloc.Object",
@@ -114,21 +150,56 @@ static PyTypeObject TallocObject_Type = {
.tp_dealloc = (destructor)pytalloc_dealloc,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
.tp_repr = pytalloc_default_repr,
+#if PY_MAJOR_VERSION >= 3
+ .tp_richcompare = pytalloc_default_richcmp,
+#else
.tp_compare = pytalloc_default_cmp,
+#endif
};
-void inittalloc(void)
+#define MODULE_DOC PyDoc_STR("Python wrapping of talloc-maintained objects.")
+
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef moduledef = {
+ PyModuleDef_HEAD_INIT,
+ .m_name = "talloc",
+ .m_doc = MODULE_DOC,
+ .m_size = -1,
+ .m_methods = talloc_methods,
+};
+#endif
+
+static PyObject *module_init(void);
+static PyObject *module_init(void)
{
PyObject *m;
if (PyType_Ready(&TallocObject_Type) < 0)
- return;
+ return NULL;
- m = Py_InitModule3("talloc", talloc_methods,
- "Python wrapping of talloc-maintained objects.");
+#if PY_MAJOR_VERSION >= 3
+ m = PyModule_Create(&moduledef);
+#else
+ m = Py_InitModule3("talloc", talloc_methods, MODULE_DOC);
+#endif
if (m == NULL)
- return;
+ return NULL;
Py_INCREF(&TallocObject_Type);
PyModule_AddObject(m, "Object", (PyObject *)&TallocObject_Type);
+ return m;
+}
+
+#if PY_MAJOR_VERSION >= 3
+PyMODINIT_FUNC PyInit_talloc(void);
+PyMODINIT_FUNC PyInit_talloc(void)
+{
+ return module_init();
+}
+#else
+void inittalloc(void);
+void inittalloc(void)
+{
+ module_init();
}
+#endif
diff --git a/lib/talloc/pytalloc.h b/lib/talloc/pytalloc.h
index 5c3876e..608328e 100644
--- a/lib/talloc/pytalloc.h
+++ b/lib/talloc/pytalloc.h
@@ -52,6 +52,8 @@ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_ctx, void
#define pytalloc_new(type, typeobj) pytalloc_steal(typeobj, talloc_zero(NULL, type))
+#if PY_MAJOR_VERSION < 3
PyObject *pytalloc_CObject_FromTallocPtr(void *);
+#endif
#endif /* _PYTALLOC_H_ */
diff --git a/lib/talloc/pytalloc_guide.txt b/lib/talloc/pytalloc_guide.txt
index 755a52b..36ae5ff 100644
--- a/lib/talloc/pytalloc_guide.txt
+++ b/lib/talloc/pytalloc_guide.txt
@@ -20,6 +20,14 @@ for objects that wrap talloc-maintained memory in C. It won't write your
bindings for you but it will make it easier to write C bindings that involve
talloc, and take away some of the boiler plate.
+Python 3
+--------
+
+pytalloc can be used with Python 3. Usage from Python extension remains
+the same, but for the C utilities, the library to link to is tagged with
+Python's PEP3149 ABI tag, for example "pytalloc.cpython34m".
+To make a build for Python 3, configure with PYTHON=/usr/bin/python3.
+.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
pytalloc_Object
@@ -126,6 +134,8 @@ use a generic VoidPtr Python type, which just provides an opaque object in
Python. The caller is responsible for incrementing the talloc reference count before calling
this function - it will dereference the talloc pointer when it is garbage collected.
+This function is only available on Python 2.
+
Debug function for talloc in Python
-----------------------------------
diff --git a/lib/talloc/pytalloc_util.c b/lib/talloc/pytalloc_util.c
index 89a093b..0af7c05 100644
--- a/lib/talloc/pytalloc_util.c
+++ b/lib/talloc/pytalloc_util.c
@@ -97,6 +97,8 @@ _PUBLIC_ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_
return (PyObject *)ret;
}
+#if PY_MAJOR_VERSION < 3
+
static void py_cobject_talloc_free(void *ptr)
{
talloc_free(ptr);
@@ -110,6 +112,8 @@ _PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
}
+#endif
+
_PUBLIC_ int pytalloc_Check(PyObject *obj)
{
PyTypeObject *tp = pytalloc_GetObjectType();
diff --git a/lib/talloc/test_pytalloc.c b/lib/talloc/test_pytalloc.c
index 2eaa7c3..f66b4e5 100644
--- a/lib/talloc/test_pytalloc.c
+++ b/lib/talloc/test_pytalloc.c
@@ -104,25 +104,56 @@ static PyTypeObject DObject_Type = {
.tp_doc = "test talloc object that calls a function when underlying data is freed\n",
};
-#define MODULE_DOC "Test utility module for pytalloc"
+#define MODULE_DOC PyDoc_STR("Test utility module for pytalloc")
+
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef moduledef = {
+ PyModuleDef_HEAD_INIT,
+ .m_name = "_test_pytalloc",
+ .m_doc = PyDoc_STR("Test utility module for pytalloc"),
+ .m_size = -1,
+ .m_methods = test_talloc_methods,
+};
+#endif
-void init_test_pytalloc(void);
-void init_test_pytalloc(void)
+static PyObject *module_init(void);
+static PyObject *module_init(void)
{
PyObject *m;
DObject_Type.tp_base = pytalloc_GetObjectType();
if (PyType_Ready(&DObject_Type) < 0) {
- return;
+ return NULL;
}
+#if PY_MAJOR_VERSION >= 3
+ m = PyModule_Create(&moduledef);
+#else
m = Py_InitModule3("_test_pytalloc", test_talloc_methods, MODULE_DOC);
+#endif
if (m == NULL) {
- return;
+ return NULL;
}
Py_INCREF(&DObject_Type);
Py_INCREF(DObject_Type.tp_base);
PyModule_AddObject(m, "DObject", (PyObject *)&DObject_Type);
+
+ return m;
+}
+
+
+#if PY_MAJOR_VERSION >= 3
+PyMODINIT_FUNC PyInit__test_pytalloc(void);
+PyMODINIT_FUNC PyInit__test_pytalloc(void)
+{
+ return module_init();
+}
+#else
+void init_test_pytalloc(void);
+void init_test_pytalloc(void)
+{
+ module_init();
}
+#endif
diff --git a/lib/talloc/test_pytalloc.py b/lib/talloc/test_pytalloc.py
index 961bfcb..a613373 100644
--- a/lib/talloc/test_pytalloc.py
+++ b/lib/talloc/test_pytalloc.py
@@ -78,6 +78,9 @@ class TallocComparisonTests(unittest.TestCase):
def test_compare_different_types(self):
# object comparison falls back to comparing types
+ if sys.version_info >= (3, 0):
+ # In Python 3, types are unorderable -- nothing to test
+ return
if talloc.Object < _test_pytalloc.DObject:
obj1 = _test_pytalloc.new()
obj2 = _test_pytalloc.DObject(dummy_func)
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 1367988..3bc932e 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -60,9 +60,7 @@ def configure(conf):
if not conf.env.disable_python:
# also disable if we don't have the python libs installed
- conf.find_program('python', var='PYTHON')
- conf.check_tool('python')
- conf.check_python_version((2,4,2))
+ conf.SAMBA_CHECK_PYTHON(mandatory=False, version=(2,4,2))
conf.SAMBA_CHECK_PYTHON_HEADERS(mandatory=False)
if not conf.env.HAVE_PYTHON_H:
Logs.warn('Disabling pytalloc-util as python devel libs not found')
@@ -118,7 +116,9 @@ def build(bld):
manpages='man/talloc.3')
if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
- bld.SAMBA_LIBRARY('pytalloc-util',
+ name = bld.pyembed_libname('pytalloc-util')
+
+ bld.SAMBA_LIBRARY(name,
source='pytalloc_util.c',
public_deps='talloc',
pyembed=True,
@@ -132,7 +132,7 @@ def build(bld):
)
bld.SAMBA_PYTHON('pytalloc',
'pytalloc.c',
- deps='talloc pytalloc-util',
+ deps='talloc ' + name,
enabled=True,
realname='talloc.so')
--
2.1.0
From c1c0d13e119e4323f6cd999a45af4db03dde1ce1 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 12:45:42 +0200
Subject: [PATCH 2/7] buildtools: Expose the Python 3 ABI tag
Expose the tag in the env to allow using it in pkg-config files
Add a "pyembed_libname" function to correctly form library names.
(This can't be done automatically in SAMBA_LIBRARY because the name
could be used as a dependency for another library)
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
buildtools/wafsamba/samba_python.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index a371b43..b90655d 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -23,6 +23,12 @@ def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
else:
conf.msg("python headers", "using cache")
+ if conf.env['PYTHON_VERSION'] > '3':
+ abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
+ conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
+ else:
+ conf.env['PYTHON_SO_ABI_FLAG'] = ''
+
def SAMBA_PYTHON(bld, name,
source='',
@@ -69,3 +75,9 @@ def SAMBA_PYTHON(bld, name,
enabled=enabled)
Build.BuildContext.SAMBA_PYTHON = SAMBA_PYTHON
+
+
+def pyembed_libname(bld, name, extrapython=False):
+ return name + bld.env['PYTHON_SO_ABI_FLAG']
+
+Build.BuildContext.pyembed_libname = pyembed_libname
--
2.1.0
From cc09d48a3a234e23f114654bb6ed10d90337964a Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Thu, 15 Jan 2015 14:22:22 +0100
Subject: [PATCH 3/7] buildtools: Add --extra-python configure option
This allows building Python support for two different Python versions
at the same time.
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
buildtools/wafsamba/samba_install.py | 8 +++++++-
buildtools/wafsamba/samba_python.py | 32 +++++++++++++++++++++++++++++++-
buildtools/wafsamba/wafsamba.py | 4 ++--
buildtools/wafsamba/wscript | 8 ++++++++
4 files changed, 48 insertions(+), 4 deletions(-)
diff --git a/buildtools/wafsamba/samba_install.py b/buildtools/wafsamba/samba_install.py
index aa7f143..af8d2ad 100644
--- a/buildtools/wafsamba/samba_install.py
+++ b/buildtools/wafsamba/samba_install.py
@@ -59,6 +59,10 @@ def install_library(self):
bld = self.bld
+ default_env = bld.all_envs['default']
+ if self.env['IS_EXTRA_PYTHON']:
+ bld.all_envs['default'] = bld.all_envs['extrapython']
+
install_ldflags = install_rpath(self)
build_ldflags = build_rpath(bld)
@@ -83,7 +87,7 @@ def install_library(self):
# install link. That stops us from overwriting the existing build
# target, which has different ldflags
self.done_install_library = True
- t = self.clone('default')
+ t = self.clone(self.env)
t.posted = False
t.target += '.inst'
self.env.RPATH = build_ldflags
@@ -144,6 +148,8 @@ def install_library(self):
if dev_link:
bld.symlink_as(os.path.join(install_path, dev_link), os.path.basename(install_name))
+ bld.all_envs['default'] = default_env
+
@feature('cshlib')
@after('apply_implib')
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index b90655d..d12fe95 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -9,20 +9,50 @@ from Configure import conf
@conf
def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
# enable tool to build python extensions
+ if conf.env['EXTRA_PYTHON']:
+ conf.all_envs['extrapython'] = conf.env.copy()
+ conf.setenv('extrapython')
+ conf.env['PYTHON'] = conf.env['EXTRA_PYTHON']
+ conf.env['IS_EXTRA_PYTHON'] = 'yes'
+ conf.find_program('python', var='PYTHON', mandatory=True)
+ conf.check_tool('python')
+ try:
+ conf.check_python_version((3, 3, 0))
+ except Exception:
+ warn('extra-python needs to be Python 3.3 or later')
+ raise
+ conf.setenv('default')
+
conf.find_program('python', var='PYTHON', mandatory=mandatory)
conf.check_tool('python')
path_python = conf.find_program('python')
conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
conf.check_python_version(version)
+
@conf
def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
if conf.env["python_headers_checked"] == []:
- conf.check_python_headers(mandatory)
+ if conf.env['EXTRA_PYTHON']:
+ conf.setenv('extrapython')
+ _check_python_headers(conf, mandatory=True)
+ conf.setenv('default')
+
+ _check_python_headers(conf, mandatory)
conf.env["python_headers_checked"] = "yes"
+
+ if conf.env['EXTRA_PYTHON']:
+ extraversion = conf.all_envs['extrapython']['PYTHON_VERSION']
+ if extraversion == conf.env['PYTHON_VERSION']:
+ raise Utils.WafError("extrapython %s is same as main python %s" % (
+ extraversion, conf.env['PYTHON_VERSION']))
else:
conf.msg("python headers", "using cache")
+
+def _check_python_headers(conf, mandatory):
+ conf.check_python_headers(mandatory=mandatory)
+
if conf.env['PYTHON_VERSION'] > '3':
abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
index d7e482c..12bf231 100644
--- a/buildtools/wafsamba/wafsamba.py
+++ b/buildtools/wafsamba/wafsamba.py
@@ -217,10 +217,10 @@ def SAMBA_LIBRARY(bld, libname, source,
if vnum is None and soname is None:
raise Utils.WafError("public library '%s' must have a vnum" %
libname)
- if pc_files is None:
+ if pc_files is None and not bld.env['IS_EXTRA_PYTHON']:
raise Utils.WafError("public library '%s' must have pkg-config file" %
libname)
- if public_headers is None:
+ if public_headers is None and not bld.env['IS_EXTRA_PYTHON']:
raise Utils.WafError("public library '%s' must have header files" %
libname)
diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript
index 694147e..d6bb688 100755
--- a/buildtools/wafsamba/wscript
+++ b/buildtools/wafsamba/wscript
@@ -195,6 +195,12 @@ def set_options(opt):
help='tag release in git at the same time',
type='string', action='store', dest='TAG_RELEASE')
+ opt.add_option('--extra-python', type=str,
+ help=("build selected libraries for the specified "
+ "additional version of Python "
+ "(example: --extra-python=/usr/bin/python3)"),
+ metavar="PYTHON", dest='EXTRA_PYTHON', default=None)
+
@wafsamba.runonce
def configure(conf):
@@ -266,6 +272,8 @@ def configure(conf):
conf.env.AUTOCONF_HOST = Options.options.AUTOCONF_HOST
conf.env.AUTOCONF_PROGRAM_PREFIX = Options.options.AUTOCONF_PROGRAM_PREFIX
+ conf.env.EXTRA_PYTHON = Options.options.EXTRA_PYTHON
+
if (conf.env.AUTOCONF_HOST and
conf.env.AUTOCONF_BUILD and
conf.env.AUTOCONF_BUILD != conf.env.AUTOCONF_HOST):
--
2.1.0
From 078383449579ca658312f6a69b15baf6135f33a5 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 18:17:06 +0200
Subject: [PATCH 4/7] pytalloc: Build for two Python versions at once
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
lib/talloc/wscript | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 3bc932e..0b62d40 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -143,6 +143,43 @@ def build(bld):
realname='_test_pytalloc.so',
install=False)
+ if bld.env['EXTRA_PYTHON']:
+ for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
+ bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
+ bak = bld.all_envs['default']
+ bld.all_envs['default'] = bld.all_envs['extrapython']
+
+ name = bld.pyembed_libname('pytalloc-util')
+
+ bld.SAMBA_LIBRARY(name,
+ source='pytalloc_util.c',
+ public_deps='talloc',
+ pyembed=True,
+ vnum=VERSION,
+ hide_symbols=True,
+ abi_directory='ABI',
+ abi_match='pytalloc_*',
+ private_library=private_library,
+ #public_headers='pytalloc.h',
+ #pc_files='pytalloc-util.pc'
+ )
+
+ bld.SAMBA_PYTHON('extra-pytalloc',
+ 'pytalloc.c',
+ deps='talloc ' + name,
+ enabled=True,
+ realname='talloc.so')
+
+ bld.SAMBA_PYTHON('extra-test_pytalloc',
+ 'test_pytalloc.c',
+ deps='pytalloc',
+ enabled=True,
+ realname='_test_pytalloc.so',
+ install=False)
+
+ bld.all_envs['default'] = bak
+
+
def test(ctx):
'''run talloc testsuite'''
import Utils, samba_utils
--
2.1.0
From b52eec2f7446888ed5812f7c0b30612f06adfca0 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 17:50:57 +0200
Subject: [PATCH 5/7] buildtools: Add a helper to iterate through Python
environments
This prevents code duplication to ensure the "extrapython" build
is the same as the normal one.
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
buildtools/wafsamba/samba_python.py | 25 +++++++++++
buildtools/wafsamba/wafsamba.py | 3 ++
lib/talloc/wscript | 89 +++++++++++--------------------------
3 files changed, 55 insertions(+), 62 deletions(-)
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index d12fe95..a0e0ffa 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -74,6 +74,9 @@ def SAMBA_PYTHON(bld, name,
enabled=True):
'''build a python extension for Samba'''
+ if bld.env['IS_EXTRA_PYTHON']:
+ name = 'extra-' + name
+
# when we support static python modules we'll need to gather
# the list from all the SAMBA_PYTHON() targets
if init_function_sentinel is not None:
@@ -111,3 +114,25 @@ def pyembed_libname(bld, name, extrapython=False):
return name + bld.env['PYTHON_SO_ABI_FLAG']
Build.BuildContext.pyembed_libname = pyembed_libname
+
+
+def gen_python_environments(bld, extra_env_vars=()):
+ """Generate all Python environments
+
+ To be used in a for loop. Normally, the loop body will be executed once.
+
+ When --extra-python is used, the body will additionaly be executed
+ with the extra-python environment active.
+ """
+ yield
+
+ if bld.env['EXTRA_PYTHON']:
+ copied = ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE') + tuple(extra_env_vars)
+ for name in copied:
+ bld.all_envs['extrapython'][name] = bld.all_envs['default'][name]
+ default_env = bld.all_envs['default']
+ bld.all_envs['default'] = bld.all_envs['extrapython']
+ yield
+ bld.all_envs['default'] = default_env
+
+Build.BuildContext.gen_python_environments = gen_python_environments
diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
index 12bf231..64382da 100644
--- a/buildtools/wafsamba/wafsamba.py
+++ b/buildtools/wafsamba/wafsamba.py
@@ -143,6 +143,9 @@ def SAMBA_LIBRARY(bld, libname, source,
enabled=True):
'''define a Samba library'''
+ if pyembed and bld.env['IS_EXTRA_PYTHON']:
+ public_headers = pc_files = None
+
if LIB_MUST_BE_PRIVATE(bld, libname):
private_library=True
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index 0b62d40..c520294 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -116,68 +116,33 @@ def build(bld):
manpages='man/talloc.3')
if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
- name = bld.pyembed_libname('pytalloc-util')
-
- bld.SAMBA_LIBRARY(name,
- source='pytalloc_util.c',
- public_deps='talloc',
- pyembed=True,
- vnum=VERSION,
- hide_symbols=True,
- abi_directory='ABI',
- abi_match='pytalloc_*',
- private_library=private_library,
- public_headers='pytalloc.h',
- pc_files='pytalloc-util.pc'
- )
- bld.SAMBA_PYTHON('pytalloc',
- 'pytalloc.c',
- deps='talloc ' + name,
- enabled=True,
- realname='talloc.so')
-
- bld.SAMBA_PYTHON('test_pytalloc',
- 'test_pytalloc.c',
- deps='pytalloc',
- enabled=True,
- realname='_test_pytalloc.so',
- install=False)
-
- if bld.env['EXTRA_PYTHON']:
- for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
- bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
- bak = bld.all_envs['default']
- bld.all_envs['default'] = bld.all_envs['extrapython']
-
- name = bld.pyembed_libname('pytalloc-util')
-
- bld.SAMBA_LIBRARY(name,
- source='pytalloc_util.c',
- public_deps='talloc',
- pyembed=True,
- vnum=VERSION,
- hide_symbols=True,
- abi_directory='ABI',
- abi_match='pytalloc_*',
- private_library=private_library,
- #public_headers='pytalloc.h',
- #pc_files='pytalloc-util.pc'
- )
-
- bld.SAMBA_PYTHON('extra-pytalloc',
- 'pytalloc.c',
- deps='talloc ' + name,
- enabled=True,
- realname='talloc.so')
-
- bld.SAMBA_PYTHON('extra-test_pytalloc',
- 'test_pytalloc.c',
- deps='pytalloc',
- enabled=True,
- realname='_test_pytalloc.so',
- install=False)
-
- bld.all_envs['default'] = bak
+ for env in bld.gen_python_environments(['PKGCONFIGDIR']):
+ name = bld.pyembed_libname('pytalloc-util')
+
+ bld.SAMBA_LIBRARY(name,
+ source='pytalloc_util.c',
+ public_deps='talloc',
+ pyembed=True,
+ vnum=VERSION,
+ hide_symbols=True,
+ abi_directory='ABI',
+ abi_match='pytalloc_*',
+ private_library=private_library,
+ public_headers='pytalloc.h',
+ pc_files='pytalloc-util.pc'
+ )
+ bld.SAMBA_PYTHON('pytalloc',
+ 'pytalloc.c',
+ deps='talloc ' + name,
+ enabled=True,
+ realname='talloc.so')
+
+ bld.SAMBA_PYTHON('test_pytalloc',
+ 'test_pytalloc.c',
+ deps='pytalloc',
+ enabled=True,
+ realname='_test_pytalloc.so',
+ install=False)
def test(ctx):
--
2.1.0
From c5fb718a8aabaf8434ad5a0d48436affb0d2d63a Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Tue, 10 Mar 2015 18:19:14 +0100
Subject: [PATCH 6/7] buildtools: Add a helper for running Python tests
Add the function samba_utils.RUN_PYTHON_TESTS for running a Python
test. When building for multiple Python versions, all are tested.
Also, add the list of configured Python interpreters to build config.
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
buildtools/wafsamba/samba_python.py | 6 ++++++
buildtools/wafsamba/samba_utils.py | 16 ++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index a0e0ffa..8b20066 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -9,6 +9,8 @@ from Configure import conf
@conf
def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
# enable tool to build python extensions
+ interpreters = []
+
if conf.env['EXTRA_PYTHON']:
conf.all_envs['extrapython'] = conf.env.copy()
conf.setenv('extrapython')
@@ -21,6 +23,7 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
except Exception:
warn('extra-python needs to be Python 3.3 or later')
raise
+ interpreters.append(conf.env['PYTHON'])
conf.setenv('default')
conf.find_program('python', var='PYTHON', mandatory=mandatory)
@@ -29,6 +32,9 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
conf.check_python_version(version)
+ interpreters.append(conf.env['PYTHON'])
+ conf.env.python_interpreters = interpreters
+
@conf
def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
diff --git a/buildtools/wafsamba/samba_utils.py b/buildtools/wafsamba/samba_utils.py
index e8bc0f3..540fe44 100644
--- a/buildtools/wafsamba/samba_utils.py
+++ b/buildtools/wafsamba/samba_utils.py
@@ -386,6 +386,22 @@ def RUN_COMMAND(cmd,
return -1
+def RUN_PYTHON_TESTS(testfiles, pythonpath=None):
+ env = LOAD_ENVIRONMENT()
+ if pythonpath is None:
+ pythonpath = os.path.join(Utils.g_module.blddir, 'python')
+ result = 0
+ for interp in env.python_interpreters:
+ for testfile in testfiles:
+ cmd = "PYTHONPATH=%s %s %s" % (pythonpath, interp, testfile)
+ print('Running Python test with %s: %s' % (interp, testfile))
+ ret = RUN_COMMAND(cmd)
+ if ret:
+ print('Python test failed: %s' % cmd)
+ result = ret
+ return result
+
+
# make sure we have md5. some systems don't have it
try:
from hashlib import md5
--
2.1.0
From d22b7758d07180e9755a07948751bf4c216c29ff Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori at redhat.com>
Date: Wed, 6 May 2015 18:05:18 +0200
Subject: [PATCH 7/7] pytalloc: Test for all Python versions
Signed-off-by: Petr Viktorin <pviktori at redhat.com>
---
lib/talloc/wscript | 10 +---------
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/lib/talloc/wscript b/lib/talloc/wscript
index c520294..8e61516 100644
--- a/lib/talloc/wscript
+++ b/lib/talloc/wscript
@@ -148,18 +148,10 @@ def build(bld):
def test(ctx):
'''run talloc testsuite'''
import Utils, samba_utils
- env = samba_utils.LOAD_ENVIRONMENT()
cmd = os.path.join(Utils.g_module.blddir, 'talloc_testsuite')
ret = samba_utils.RUN_COMMAND(cmd)
print("testsuite returned %d" % ret)
- if 'USING_SYSTEM_PYTALLOC_UTIL' not in env.defines and not env.disable_python:
- cmd = "PYTHONPATH=%s %s test_pytalloc.py" % (
- os.path.join(Utils.g_module.blddir, 'python'),
- env['PYTHON'],
- )
- pyret = samba_utils.RUN_COMMAND(cmd)
- else:
- pyret = 0
+ pyret = samba_utils.RUN_PYTHON_TESTS(['test_pytalloc.py'])
print("python testsuite returned %d" % pyret)
sys.exit(ret or pyret)
--
2.1.0
More information about the samba-technical
mailing list