pytalloc patches
Jelmer Vernooij
jelmer at samba.org
Mon May 11 08:42:04 MDT 2015
On Mon, May 11, 2015 at 02:56:15PM +0200, Petr Viktorin wrote:
> After some off-list discussion with Thomas Nagy about modifications to
> waf, I am submitting reworked buildsystem/pytalloc patches for another
> round of review.
> This version works with a copy of the build environment, so the changes
> are less invasive.
Thanks, Petr! This looks much nicer.
One quick question: why does the EXTRAPYTHON need to be version 3.3 or later?
Would it not be possible to build for python2.6 and 2.7 (say) ?
Andrew, any chance you can also look over these and give your Reviewed-By?
Cheers,
Jelmer
> With Thomas' permission, I'm forwarding the discussion:
> Thomas Nagy wrote:
> >
> > The latest changes provide a much better separation of concerns, and they are easier to review. They are mostly fine.
> >
> > One suggestion for future changes:
> > + for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
> > + bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
> > The best practice is to adjust environment settings during the configuration phase so that:
> > * duplication of initialization code is prevented
> > * accidental dependencies between build scripts are not created (ideally, build scripts could be read in any particular order)
> > * analysis and debugging is simplified (tweak variables without any project reconfiguration)
> >
> > When environment objects differ too much to be merged, it is possible to:
> > * override the method ConfigurationContext.store() to perform last-minute adjustments
> > * or to use a global bld.add_pre_fun(function) to perform a global initialization
> >
> > Thomas
> >
> > On Wed, 06 May 2015 19:23:41 +0200, Petr Viktorin wrote:
> >> On 05/04/2015 08:57 PM, Thomas Nagy wrote:
> >>> On Mon, 04 May 2015 17:02:03 +0200, Petr Viktorin wrote:
> >>>
> >>>> On 05/03/2015 09:19 PM, T N wrote:
> >>>>> The patches above may work, but what would happen if a yet another
> >>>>> python version became necessary?
> >>>>
> >>>> As in building for *three* different versions of Python? It would still
> >>>> be possible, just not simultaneously. Build for two, then reconfigure
> >>>> and build for the third.
> >>>>
> >>>> I agree that my solution is not general in this respect, but I don't
> >>>> think it's a practical concern.
> >>>>
> >>>>> Since the build holds all the data in
> >>>>> the conf.env/bld.env objects, it may be more simple to either:
> >>>>> * Set another conf.env/bld.env before declaring tests/targets, without
> >>>>> swapping variables manually and without additional variables. For
> >>>>> example, to have all targets in a particular file use that particular
> >>>>> interpreter, set bld.env=bld.all_envs['fancy_extra_python'] on top of
> >>>>> the file and reset it with bld.env=bld.all_envs['default'] at the end.
> >>>>
> >>>> Well, Build.env is defined as:
> >>>>
> >>>> def get_env(self):
> >>>> return self.env_of_name('default')
> >>>> def set_env(self, name, val):
> >>>> self.all_envs[name] = val
> >>>>
> >>>> env = property(get_env, set_env)
> >>>>
> >>>> So `bld.env=X` calls `set_env(bld, X)`, which fails with wrong number of
> >>>> arguments. Even if it did work, bld.env is hard-wired to be
> >>>> bld.env_of_name('default').
> >>>
> >>> And what would be the problem if it worked? Are the targets using that python version grouped together in the same wscript file? in this case - in pseudocode - it would suffice to use the other env object in the file:
> >>>
> >>> """
> >>> bak = bld.env
> >>> bld.all_envs['default'] = bld.all_envs['extrapython']
> >>>
> >>> bld.SAMBA_TARGET(...)
> >>> bld.SAMBA_LIBRARY(...)
> >>>
> >>> bld.all_envs['default'] = bak
> >>> """
> >>>
> >>> and the environment can be prepared during the configuration through:
> >>> """
> >>> env = conf.all_envs['extrapython'] = conf.env.copy()
> >>> env.EXTRAPYTHON = '/foo'
> >>> """
> >>
> >> A problem is that the copy is only linked to its parent during the
> >> configure step. For the build step, it's deserialized from a file, so
> >> any changes to the default env aren't reflected in the copy.
> >> Samba records the dependencies in the env. So if I have:
> >>
> >> bld.SAMBA_LIBRARY('talloc', ...)
> >>
> >> bld.all_envs['default'] = bld.all_envs['extrapython']
> >>
> >> bld.SAMBA_LIBRARY(..., public_deps='talloc')
> >>
> >> where "talloc" is not Python-specific (and thus shared between the two
> >> versions of the bindings), the dependency is not found.
> >>
> >> So I'd need to copy a bunch of variables back to the extrapython env
> >> when choosing it. There's a lot fewer of them than in the list in the
> >> patches ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR' are the
> >> ones I found), but they're not Python-specific, and would have to be
> >> updated when something outside of the Python support adds one.
> >>
> >> That worries me a bit, but it's definitely a less invasive approach.
> >>
> >>> Or is it that the targets are spread over several files?
> >>
> >> Yes, eventually each standalone library will need its own bindings. But
> >> I don't think that's a problem.
> >>
> >>>>> This may also provide a better separation of concern by avoiding the
> >>>>> addition of new parameters to all the functions above.
> >>>>> * Or to just update bld.env with the desired variables once. For example
> >>>>> in init_extrapyext: self.env=self.env.copy;self.env.table.update(...) ).
> >>>>> Here too a specific environment object can be prepared during the
> >>>>> configuration.
> >>>>
> >>>> Same problem here; I can't modify self.env (i.e.
> >>>> self.env_of_name('default')) without affecting all other uses of the env.
> >>>
> >>> The object "self.env" is a copy of bld.env, and it is definitely meant to be modified (use the methods append_value/append_unique). Each task generator is holding a copy (shallow copy), and each task object is also holding copy of the task generator env copy.
> >>>
> >>> Am I misunderstanding the intention, i.e. the code is actually merging object code of different python versions into the same libraries/programs?
> [...]
> >> I'm attaching a reworked set of patches. You can ignore the first one
> >> for now, it's not related to the build system.
> >> Does this look better to you?
> >>
> >> (I'm not yet convinced on the 5th patch, "Add a helper to iterate
> >> through Python": it reduces boilerplate, but perhaps it's too magic to
> >> have a for loop's iterator with side effects. But that's just a detail.)
>
> From eb70ce83a9f8d4ce7e1b10cc649d412733c6704f Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Thu, 15 Jan 2015 14:07:09 +0100
> Subject: [PATCH 1/7] pytalloc: Port to Python 3
>
> - Use native string for repr
>
> - Use rich comparison
> Removes the deprecated tp_compare in favor of tp_richcompare.
> Disparate types cannot be compared (except for == and !=),
> and True or False objects are returned explicitly.
>
> - Use Py_TYPE instead of ob_type
> This changed to conform to C aliasing rules,
> see http://legacy.python.org/dev/peps/pep-3123/
>
> - Don't provide CObject creation function
> A PyCapsule based replacement would be possible,
> but might not be necessary considering the function is
> not used much.
>
> - Use new-style module initialization
>
> Build changes:
>
> - Use ABI flag in the lib name and pkg-config template
>
> - Use the SAMBA_CHECK_PYTHON macro for finding Python
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> lib/talloc/pytalloc-util.pc.in | 2 +-
> lib/talloc/pytalloc.c | 87 ++++++++++++++++++++++++++++++++++++++----
> lib/talloc/pytalloc.h | 2 +
> lib/talloc/pytalloc_guide.txt | 10 +++++
> lib/talloc/pytalloc_util.c | 4 ++
> lib/talloc/test_pytalloc.c | 41 +++++++++++++++++---
> lib/talloc/test_pytalloc.py | 3 ++
> lib/talloc/wscript | 10 ++---
> 8 files changed, 140 insertions(+), 19 deletions(-)
>
> diff --git a/lib/talloc/pytalloc-util.pc.in b/lib/talloc/pytalloc-util.pc.in
> index b7426bb..b87c94e 100644
> --- a/lib/talloc/pytalloc-util.pc.in
> +++ b/lib/talloc/pytalloc-util.pc.in
> @@ -6,6 +6,6 @@ includedir=@includedir@
> Name: pytalloc-util
> Description: Utility functions for using talloc objects with Python
> Version: @TALLOC_VERSION@
> -Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util
> +Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-util at PYTHON_SO_ABI_FLAG@
> Cflags: -I${includedir}
> URL: http://talloc.samba.org/
> diff --git a/lib/talloc/pytalloc.c b/lib/talloc/pytalloc.c
> index ac4fe0f..3afae9c 100644
> --- a/lib/talloc/pytalloc.c
> +++ b/lib/talloc/pytalloc.c
> @@ -21,7 +21,13 @@
> #include <talloc.h>
> #include <pytalloc.h>
>
> -void inittalloc(void);
> +static PyTypeObject TallocObject_Type;
> +
> +#if PY_MAJOR_VERSION >= 3
> +#define PyStr_FromFormat PyUnicode_FromFormat
> +#else
> +#define PyStr_FromFormat PyString_FromFormat
> +#endif
>
> /* print a talloc tree report for a talloc python object */
> static PyObject *pytalloc_report_full(PyObject *self, PyObject *args)
> @@ -79,8 +85,8 @@ static PyObject *pytalloc_default_repr(PyObject *obj)
> pytalloc_Object *talloc_obj = (pytalloc_Object *)obj;
> PyTypeObject *type = (PyTypeObject*)PyObject_Type(obj);
>
> - return PyString_FromFormat("<%s talloc object at 0x%p>",
> - type->tp_name, talloc_obj->ptr);
> + return PyStr_FromFormat("<%s talloc object at 0x%p>",
> + type->tp_name, talloc_obj->ptr);
> }
>
> /**
> @@ -97,6 +103,35 @@ static void pytalloc_dealloc(PyObject* self)
> /**
> * Default (but only slightly more useful than the default) implementation of cmp.
> */
> +#if PY_MAJOR_VERSION >= 3
> +static PyObject *pytalloc_default_richcmp(PyObject *obj1, PyObject *obj2, int op)
> +{
> + void *ptr1;
> + void *ptr2;
> + if (Py_TYPE(obj1) == Py_TYPE(obj2)) {
> + /* When types match, compare pointers */
> + ptr1 = pytalloc_get_ptr(obj1);
> + ptr2 = pytalloc_get_ptr(obj2);
> + } else if (PyObject_TypeCheck(obj2, &TallocObject_Type)) {
> + /* Otherwise, compare types */
> + ptr1 = Py_TYPE(obj1);
> + ptr2 = Py_TYPE(obj2);
> + } else {
> + Py_INCREF(Py_NotImplemented);
> + return Py_NotImplemented;
> + }
> + switch (op) {
> + case Py_EQ: return PyBool_FromLong(ptr1 == ptr2);
> + case Py_NE: return PyBool_FromLong(ptr1 != ptr2);
> + case Py_LT: return PyBool_FromLong(ptr1 < ptr2);
> + case Py_GT: return PyBool_FromLong(ptr1 > ptr2);
> + case Py_LE: return PyBool_FromLong(ptr1 <= ptr2);
> + case Py_GE: return PyBool_FromLong(ptr1 >= ptr2);
> + }
> + Py_INCREF(Py_NotImplemented);
> + return Py_NotImplemented;
> +}
> +#else
> static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
> {
> pytalloc_Object *obj1 = (pytalloc_Object *)_obj1,
> @@ -106,6 +141,7 @@ static int pytalloc_default_cmp(PyObject *_obj1, PyObject *_obj2)
>
> return ((char *)pytalloc_get_ptr(obj1) - (char *)pytalloc_get_ptr(obj2));
> }
> +#endif
>
> static PyTypeObject TallocObject_Type = {
> .tp_name = "talloc.Object",
> @@ -114,21 +150,56 @@ static PyTypeObject TallocObject_Type = {
> .tp_dealloc = (destructor)pytalloc_dealloc,
> .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
> .tp_repr = pytalloc_default_repr,
> +#if PY_MAJOR_VERSION >= 3
> + .tp_richcompare = pytalloc_default_richcmp,
> +#else
> .tp_compare = pytalloc_default_cmp,
> +#endif
> };
>
> -void inittalloc(void)
> +#define MODULE_DOC PyDoc_STR("Python wrapping of talloc-maintained objects.")
> +
> +#if PY_MAJOR_VERSION >= 3
> +static struct PyModuleDef moduledef = {
> + PyModuleDef_HEAD_INIT,
> + .m_name = "talloc",
> + .m_doc = MODULE_DOC,
> + .m_size = -1,
> + .m_methods = talloc_methods,
> +};
> +#endif
> +
> +static PyObject *module_init(void);
> +static PyObject *module_init(void)
> {
> PyObject *m;
>
> if (PyType_Ready(&TallocObject_Type) < 0)
> - return;
> + return NULL;
>
> - m = Py_InitModule3("talloc", talloc_methods,
> - "Python wrapping of talloc-maintained objects.");
> +#if PY_MAJOR_VERSION >= 3
> + m = PyModule_Create(&moduledef);
> +#else
> + m = Py_InitModule3("talloc", talloc_methods, MODULE_DOC);
> +#endif
> if (m == NULL)
> - return;
> + return NULL;
>
> Py_INCREF(&TallocObject_Type);
> PyModule_AddObject(m, "Object", (PyObject *)&TallocObject_Type);
> + return m;
> +}
> +
> +#if PY_MAJOR_VERSION >= 3
> +PyMODINIT_FUNC PyInit_talloc(void);
> +PyMODINIT_FUNC PyInit_talloc(void)
> +{
> + return module_init();
> +}
> +#else
> +void inittalloc(void);
> +void inittalloc(void)
> +{
> + module_init();
> }
> +#endif
> diff --git a/lib/talloc/pytalloc.h b/lib/talloc/pytalloc.h
> index 5c3876e..608328e 100644
> --- a/lib/talloc/pytalloc.h
> +++ b/lib/talloc/pytalloc.h
> @@ -52,6 +52,8 @@ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_ctx, void
>
> #define pytalloc_new(type, typeobj) pytalloc_steal(typeobj, talloc_zero(NULL, type))
>
> +#if PY_MAJOR_VERSION < 3
> PyObject *pytalloc_CObject_FromTallocPtr(void *);
> +#endif
>
> #endif /* _PYTALLOC_H_ */
> diff --git a/lib/talloc/pytalloc_guide.txt b/lib/talloc/pytalloc_guide.txt
> index 755a52b..36ae5ff 100644
> --- a/lib/talloc/pytalloc_guide.txt
> +++ b/lib/talloc/pytalloc_guide.txt
> @@ -20,6 +20,14 @@ for objects that wrap talloc-maintained memory in C. It won't write your
> bindings for you but it will make it easier to write C bindings that involve
> talloc, and take away some of the boiler plate.
>
> +Python 3
> +--------
> +
> +pytalloc can be used with Python 3. Usage from Python extension remains
> +the same, but for the C utilities, the library to link to is tagged with
> +Python's PEP3149 ABI tag, for example "pytalloc.cpython34m".
> +To make a build for Python 3, configure with PYTHON=/usr/bin/python3.
> +.
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> pytalloc_Object
>
> @@ -126,6 +134,8 @@ use a generic VoidPtr Python type, which just provides an opaque object in
> Python. The caller is responsible for incrementing the talloc reference count before calling
> this function - it will dereference the talloc pointer when it is garbage collected.
>
> +This function is only available on Python 2.
> +
> Debug function for talloc in Python
> -----------------------------------
>
> diff --git a/lib/talloc/pytalloc_util.c b/lib/talloc/pytalloc_util.c
> index 89a093b..0af7c05 100644
> --- a/lib/talloc/pytalloc_util.c
> +++ b/lib/talloc/pytalloc_util.c
> @@ -97,6 +97,8 @@ _PUBLIC_ PyObject *pytalloc_reference_ex(PyTypeObject *py_type, TALLOC_CTX *mem_
> return (PyObject *)ret;
> }
>
> +#if PY_MAJOR_VERSION < 3
> +
> static void py_cobject_talloc_free(void *ptr)
> {
> talloc_free(ptr);
> @@ -110,6 +112,8 @@ _PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
> return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
> }
>
> +#endif
> +
> _PUBLIC_ int pytalloc_Check(PyObject *obj)
> {
> PyTypeObject *tp = pytalloc_GetObjectType();
> diff --git a/lib/talloc/test_pytalloc.c b/lib/talloc/test_pytalloc.c
> index 2eaa7c3..f66b4e5 100644
> --- a/lib/talloc/test_pytalloc.c
> +++ b/lib/talloc/test_pytalloc.c
> @@ -104,25 +104,56 @@ static PyTypeObject DObject_Type = {
> .tp_doc = "test talloc object that calls a function when underlying data is freed\n",
> };
>
> -#define MODULE_DOC "Test utility module for pytalloc"
> +#define MODULE_DOC PyDoc_STR("Test utility module for pytalloc")
> +
> +#if PY_MAJOR_VERSION >= 3
> +static struct PyModuleDef moduledef = {
> + PyModuleDef_HEAD_INIT,
> + .m_name = "_test_pytalloc",
> + .m_doc = PyDoc_STR("Test utility module for pytalloc"),
> + .m_size = -1,
> + .m_methods = test_talloc_methods,
> +};
> +#endif
>
> -void init_test_pytalloc(void);
> -void init_test_pytalloc(void)
> +static PyObject *module_init(void);
> +static PyObject *module_init(void)
> {
> PyObject *m;
>
> DObject_Type.tp_base = pytalloc_GetObjectType();
> if (PyType_Ready(&DObject_Type) < 0) {
> - return;
> + return NULL;
> }
>
> +#if PY_MAJOR_VERSION >= 3
> + m = PyModule_Create(&moduledef);
> +#else
> m = Py_InitModule3("_test_pytalloc", test_talloc_methods, MODULE_DOC);
> +#endif
>
> if (m == NULL) {
> - return;
> + return NULL;
> }
>
> Py_INCREF(&DObject_Type);
> Py_INCREF(DObject_Type.tp_base);
> PyModule_AddObject(m, "DObject", (PyObject *)&DObject_Type);
> +
> + return m;
> +}
> +
> +
> +#if PY_MAJOR_VERSION >= 3
> +PyMODINIT_FUNC PyInit__test_pytalloc(void);
> +PyMODINIT_FUNC PyInit__test_pytalloc(void)
> +{
> + return module_init();
> +}
> +#else
> +void init_test_pytalloc(void);
> +void init_test_pytalloc(void)
> +{
> + module_init();
> }
> +#endif
> diff --git a/lib/talloc/test_pytalloc.py b/lib/talloc/test_pytalloc.py
> index 961bfcb..a613373 100644
> --- a/lib/talloc/test_pytalloc.py
> +++ b/lib/talloc/test_pytalloc.py
> @@ -78,6 +78,9 @@ class TallocComparisonTests(unittest.TestCase):
>
> def test_compare_different_types(self):
> # object comparison falls back to comparing types
> + if sys.version_info >= (3, 0):
> + # In Python 3, types are unorderable -- nothing to test
> + return
> if talloc.Object < _test_pytalloc.DObject:
> obj1 = _test_pytalloc.new()
> obj2 = _test_pytalloc.DObject(dummy_func)
> diff --git a/lib/talloc/wscript b/lib/talloc/wscript
> index 1367988..3bc932e 100644
> --- a/lib/talloc/wscript
> +++ b/lib/talloc/wscript
> @@ -60,9 +60,7 @@ def configure(conf):
>
> if not conf.env.disable_python:
> # also disable if we don't have the python libs installed
> - conf.find_program('python', var='PYTHON')
> - conf.check_tool('python')
> - conf.check_python_version((2,4,2))
> + conf.SAMBA_CHECK_PYTHON(mandatory=False, version=(2,4,2))
> conf.SAMBA_CHECK_PYTHON_HEADERS(mandatory=False)
> if not conf.env.HAVE_PYTHON_H:
> Logs.warn('Disabling pytalloc-util as python devel libs not found')
> @@ -118,7 +116,9 @@ def build(bld):
> manpages='man/talloc.3')
>
> if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
> - bld.SAMBA_LIBRARY('pytalloc-util',
> + name = bld.pyembed_libname('pytalloc-util')
> +
> + bld.SAMBA_LIBRARY(name,
> source='pytalloc_util.c',
> public_deps='talloc',
> pyembed=True,
> @@ -132,7 +132,7 @@ def build(bld):
> )
> bld.SAMBA_PYTHON('pytalloc',
> 'pytalloc.c',
> - deps='talloc pytalloc-util',
> + deps='talloc ' + name,
> enabled=True,
> realname='talloc.so')
>
> --
> 2.1.0
>
>
> From c1c0d13e119e4323f6cd999a45af4db03dde1ce1 Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Wed, 6 May 2015 12:45:42 +0200
> Subject: [PATCH 2/7] buildtools: Expose the Python 3 ABI tag
>
> Expose the tag in the env to allow using it in pkg-config files
>
> Add a "pyembed_libname" function to correctly form library names.
> (This can't be done automatically in SAMBA_LIBRARY because the name
> could be used as a dependency for another library)
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> buildtools/wafsamba/samba_python.py | 12 ++++++++++++
> 1 file changed, 12 insertions(+)
>
> diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
> index a371b43..b90655d 100644
> --- a/buildtools/wafsamba/samba_python.py
> +++ b/buildtools/wafsamba/samba_python.py
> @@ -23,6 +23,12 @@ def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
> else:
> conf.msg("python headers", "using cache")
>
> + if conf.env['PYTHON_VERSION'] > '3':
> + abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
> + conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
> + else:
> + conf.env['PYTHON_SO_ABI_FLAG'] = ''
> +
>
> def SAMBA_PYTHON(bld, name,
> source='',
> @@ -69,3 +75,9 @@ def SAMBA_PYTHON(bld, name,
> enabled=enabled)
>
> Build.BuildContext.SAMBA_PYTHON = SAMBA_PYTHON
> +
> +
> +def pyembed_libname(bld, name, extrapython=False):
> + return name + bld.env['PYTHON_SO_ABI_FLAG']
> +
> +Build.BuildContext.pyembed_libname = pyembed_libname
> --
> 2.1.0
>
>
> From cc09d48a3a234e23f114654bb6ed10d90337964a Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Thu, 15 Jan 2015 14:22:22 +0100
> Subject: [PATCH 3/7] buildtools: Add --extra-python configure option
>
> This allows building Python support for two different Python versions
> at the same time.
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> buildtools/wafsamba/samba_install.py | 8 +++++++-
> buildtools/wafsamba/samba_python.py | 32 +++++++++++++++++++++++++++++++-
> buildtools/wafsamba/wafsamba.py | 4 ++--
> buildtools/wafsamba/wscript | 8 ++++++++
> 4 files changed, 48 insertions(+), 4 deletions(-)
>
> diff --git a/buildtools/wafsamba/samba_install.py b/buildtools/wafsamba/samba_install.py
> index aa7f143..af8d2ad 100644
> --- a/buildtools/wafsamba/samba_install.py
> +++ b/buildtools/wafsamba/samba_install.py
> @@ -59,6 +59,10 @@ def install_library(self):
>
> bld = self.bld
>
> + default_env = bld.all_envs['default']
> + if self.env['IS_EXTRA_PYTHON']:
> + bld.all_envs['default'] = bld.all_envs['extrapython']
> +
> install_ldflags = install_rpath(self)
> build_ldflags = build_rpath(bld)
>
> @@ -83,7 +87,7 @@ def install_library(self):
> # install link. That stops us from overwriting the existing build
> # target, which has different ldflags
> self.done_install_library = True
> - t = self.clone('default')
> + t = self.clone(self.env)
> t.posted = False
> t.target += '.inst'
> self.env.RPATH = build_ldflags
> @@ -144,6 +148,8 @@ def install_library(self):
> if dev_link:
> bld.symlink_as(os.path.join(install_path, dev_link), os.path.basename(install_name))
>
> + bld.all_envs['default'] = default_env
> +
>
> @feature('cshlib')
> @after('apply_implib')
> diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
> index b90655d..d12fe95 100644
> --- a/buildtools/wafsamba/samba_python.py
> +++ b/buildtools/wafsamba/samba_python.py
> @@ -9,20 +9,50 @@ from Configure import conf
> @conf
> def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
> # enable tool to build python extensions
> + if conf.env['EXTRA_PYTHON']:
> + conf.all_envs['extrapython'] = conf.env.copy()
> + conf.setenv('extrapython')
> + conf.env['PYTHON'] = conf.env['EXTRA_PYTHON']
> + conf.env['IS_EXTRA_PYTHON'] = 'yes'
> + conf.find_program('python', var='PYTHON', mandatory=True)
> + conf.check_tool('python')
> + try:
> + conf.check_python_version((3, 3, 0))
> + except Exception:
> + warn('extra-python needs to be Python 3.3 or later')
> + raise
> + conf.setenv('default')
> +
> conf.find_program('python', var='PYTHON', mandatory=mandatory)
> conf.check_tool('python')
> path_python = conf.find_program('python')
> conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
> conf.check_python_version(version)
>
> +
> @conf
> def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
> if conf.env["python_headers_checked"] == []:
> - conf.check_python_headers(mandatory)
> + if conf.env['EXTRA_PYTHON']:
> + conf.setenv('extrapython')
> + _check_python_headers(conf, mandatory=True)
> + conf.setenv('default')
> +
> + _check_python_headers(conf, mandatory)
> conf.env["python_headers_checked"] = "yes"
> +
> + if conf.env['EXTRA_PYTHON']:
> + extraversion = conf.all_envs['extrapython']['PYTHON_VERSION']
> + if extraversion == conf.env['PYTHON_VERSION']:
> + raise Utils.WafError("extrapython %s is same as main python %s" % (
> + extraversion, conf.env['PYTHON_VERSION']))
> else:
> conf.msg("python headers", "using cache")
>
> +
> +def _check_python_headers(conf, mandatory):
> + conf.check_python_headers(mandatory=mandatory)
> +
> if conf.env['PYTHON_VERSION'] > '3':
> abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
> conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
> diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
> index d7e482c..12bf231 100644
> --- a/buildtools/wafsamba/wafsamba.py
> +++ b/buildtools/wafsamba/wafsamba.py
> @@ -217,10 +217,10 @@ def SAMBA_LIBRARY(bld, libname, source,
> if vnum is None and soname is None:
> raise Utils.WafError("public library '%s' must have a vnum" %
> libname)
> - if pc_files is None:
> + if pc_files is None and not bld.env['IS_EXTRA_PYTHON']:
> raise Utils.WafError("public library '%s' must have pkg-config file" %
> libname)
> - if public_headers is None:
> + if public_headers is None and not bld.env['IS_EXTRA_PYTHON']:
> raise Utils.WafError("public library '%s' must have header files" %
> libname)
>
> diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript
> index 694147e..d6bb688 100755
> --- a/buildtools/wafsamba/wscript
> +++ b/buildtools/wafsamba/wscript
> @@ -195,6 +195,12 @@ def set_options(opt):
> help='tag release in git at the same time',
> type='string', action='store', dest='TAG_RELEASE')
>
> + opt.add_option('--extra-python', type=str,
> + help=("build selected libraries for the specified "
> + "additional version of Python "
> + "(example: --extra-python=/usr/bin/python3)"),
> + metavar="PYTHON", dest='EXTRA_PYTHON', default=None)
> +
>
> @wafsamba.runonce
> def configure(conf):
> @@ -266,6 +272,8 @@ def configure(conf):
> conf.env.AUTOCONF_HOST = Options.options.AUTOCONF_HOST
> conf.env.AUTOCONF_PROGRAM_PREFIX = Options.options.AUTOCONF_PROGRAM_PREFIX
>
> + conf.env.EXTRA_PYTHON = Options.options.EXTRA_PYTHON
> +
> if (conf.env.AUTOCONF_HOST and
> conf.env.AUTOCONF_BUILD and
> conf.env.AUTOCONF_BUILD != conf.env.AUTOCONF_HOST):
> --
> 2.1.0
>
>
> From 078383449579ca658312f6a69b15baf6135f33a5 Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Wed, 6 May 2015 18:17:06 +0200
> Subject: [PATCH 4/7] pytalloc: Build for two Python versions at once
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> lib/talloc/wscript | 37 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 37 insertions(+)
>
> diff --git a/lib/talloc/wscript b/lib/talloc/wscript
> index 3bc932e..0b62d40 100644
> --- a/lib/talloc/wscript
> +++ b/lib/talloc/wscript
> @@ -143,6 +143,43 @@ def build(bld):
> realname='_test_pytalloc.so',
> install=False)
>
> + if bld.env['EXTRA_PYTHON']:
> + for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
> + bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
> + bak = bld.all_envs['default']
> + bld.all_envs['default'] = bld.all_envs['extrapython']
> +
> + name = bld.pyembed_libname('pytalloc-util')
> +
> + bld.SAMBA_LIBRARY(name,
> + source='pytalloc_util.c',
> + public_deps='talloc',
> + pyembed=True,
> + vnum=VERSION,
> + hide_symbols=True,
> + abi_directory='ABI',
> + abi_match='pytalloc_*',
> + private_library=private_library,
> + #public_headers='pytalloc.h',
> + #pc_files='pytalloc-util.pc'
> + )
> +
> + bld.SAMBA_PYTHON('extra-pytalloc',
> + 'pytalloc.c',
> + deps='talloc ' + name,
> + enabled=True,
> + realname='talloc.so')
> +
> + bld.SAMBA_PYTHON('extra-test_pytalloc',
> + 'test_pytalloc.c',
> + deps='pytalloc',
> + enabled=True,
> + realname='_test_pytalloc.so',
> + install=False)
> +
> + bld.all_envs['default'] = bak
> +
> +
> def test(ctx):
> '''run talloc testsuite'''
> import Utils, samba_utils
> --
> 2.1.0
>
>
> From b52eec2f7446888ed5812f7c0b30612f06adfca0 Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Wed, 6 May 2015 17:50:57 +0200
> Subject: [PATCH 5/7] buildtools: Add a helper to iterate through Python
> environments
>
> This prevents code duplication to ensure the "extrapython" build
> is the same as the normal one.
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> buildtools/wafsamba/samba_python.py | 25 +++++++++++
> buildtools/wafsamba/wafsamba.py | 3 ++
> lib/talloc/wscript | 89 +++++++++++--------------------------
> 3 files changed, 55 insertions(+), 62 deletions(-)
>
> diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
> index d12fe95..a0e0ffa 100644
> --- a/buildtools/wafsamba/samba_python.py
> +++ b/buildtools/wafsamba/samba_python.py
> @@ -74,6 +74,9 @@ def SAMBA_PYTHON(bld, name,
> enabled=True):
> '''build a python extension for Samba'''
>
> + if bld.env['IS_EXTRA_PYTHON']:
> + name = 'extra-' + name
> +
> # when we support static python modules we'll need to gather
> # the list from all the SAMBA_PYTHON() targets
> if init_function_sentinel is not None:
> @@ -111,3 +114,25 @@ def pyembed_libname(bld, name, extrapython=False):
> return name + bld.env['PYTHON_SO_ABI_FLAG']
>
> Build.BuildContext.pyembed_libname = pyembed_libname
> +
> +
> +def gen_python_environments(bld, extra_env_vars=()):
> + """Generate all Python environments
> +
> + To be used in a for loop. Normally, the loop body will be executed once.
> +
> + When --extra-python is used, the body will additionaly be executed
> + with the extra-python environment active.
> + """
> + yield
> +
> + if bld.env['EXTRA_PYTHON']:
> + copied = ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE') + tuple(extra_env_vars)
> + for name in copied:
> + bld.all_envs['extrapython'][name] = bld.all_envs['default'][name]
> + default_env = bld.all_envs['default']
> + bld.all_envs['default'] = bld.all_envs['extrapython']
> + yield
> + bld.all_envs['default'] = default_env
> +
> +Build.BuildContext.gen_python_environments = gen_python_environments
> diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
> index 12bf231..64382da 100644
> --- a/buildtools/wafsamba/wafsamba.py
> +++ b/buildtools/wafsamba/wafsamba.py
> @@ -143,6 +143,9 @@ def SAMBA_LIBRARY(bld, libname, source,
> enabled=True):
> '''define a Samba library'''
>
> + if pyembed and bld.env['IS_EXTRA_PYTHON']:
> + public_headers = pc_files = None
> +
> if LIB_MUST_BE_PRIVATE(bld, libname):
> private_library=True
>
> diff --git a/lib/talloc/wscript b/lib/talloc/wscript
> index 0b62d40..c520294 100644
> --- a/lib/talloc/wscript
> +++ b/lib/talloc/wscript
> @@ -116,68 +116,33 @@ def build(bld):
> manpages='man/talloc.3')
>
> if not bld.CONFIG_SET('USING_SYSTEM_PYTALLOC_UTIL') and not bld.env.disable_python:
> - name = bld.pyembed_libname('pytalloc-util')
> -
> - bld.SAMBA_LIBRARY(name,
> - source='pytalloc_util.c',
> - public_deps='talloc',
> - pyembed=True,
> - vnum=VERSION,
> - hide_symbols=True,
> - abi_directory='ABI',
> - abi_match='pytalloc_*',
> - private_library=private_library,
> - public_headers='pytalloc.h',
> - pc_files='pytalloc-util.pc'
> - )
> - bld.SAMBA_PYTHON('pytalloc',
> - 'pytalloc.c',
> - deps='talloc ' + name,
> - enabled=True,
> - realname='talloc.so')
> -
> - bld.SAMBA_PYTHON('test_pytalloc',
> - 'test_pytalloc.c',
> - deps='pytalloc',
> - enabled=True,
> - realname='_test_pytalloc.so',
> - install=False)
> -
> - if bld.env['EXTRA_PYTHON']:
> - for var_name in ('GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'PKGCONFIGDIR'):
> - bld.all_envs['extrapython'][var_name] = bld.all_envs['default'][var_name]
> - bak = bld.all_envs['default']
> - bld.all_envs['default'] = bld.all_envs['extrapython']
> -
> - name = bld.pyembed_libname('pytalloc-util')
> -
> - bld.SAMBA_LIBRARY(name,
> - source='pytalloc_util.c',
> - public_deps='talloc',
> - pyembed=True,
> - vnum=VERSION,
> - hide_symbols=True,
> - abi_directory='ABI',
> - abi_match='pytalloc_*',
> - private_library=private_library,
> - #public_headers='pytalloc.h',
> - #pc_files='pytalloc-util.pc'
> - )
> -
> - bld.SAMBA_PYTHON('extra-pytalloc',
> - 'pytalloc.c',
> - deps='talloc ' + name,
> - enabled=True,
> - realname='talloc.so')
> -
> - bld.SAMBA_PYTHON('extra-test_pytalloc',
> - 'test_pytalloc.c',
> - deps='pytalloc',
> - enabled=True,
> - realname='_test_pytalloc.so',
> - install=False)
> -
> - bld.all_envs['default'] = bak
> + for env in bld.gen_python_environments(['PKGCONFIGDIR']):
> + name = bld.pyembed_libname('pytalloc-util')
> +
> + bld.SAMBA_LIBRARY(name,
> + source='pytalloc_util.c',
> + public_deps='talloc',
> + pyembed=True,
> + vnum=VERSION,
> + hide_symbols=True,
> + abi_directory='ABI',
> + abi_match='pytalloc_*',
> + private_library=private_library,
> + public_headers='pytalloc.h',
> + pc_files='pytalloc-util.pc'
> + )
> + bld.SAMBA_PYTHON('pytalloc',
> + 'pytalloc.c',
> + deps='talloc ' + name,
> + enabled=True,
> + realname='talloc.so')
> +
> + bld.SAMBA_PYTHON('test_pytalloc',
> + 'test_pytalloc.c',
> + deps='pytalloc',
> + enabled=True,
> + realname='_test_pytalloc.so',
> + install=False)
>
>
> def test(ctx):
> --
> 2.1.0
>
>
> From c5fb718a8aabaf8434ad5a0d48436affb0d2d63a Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Tue, 10 Mar 2015 18:19:14 +0100
> Subject: [PATCH 6/7] buildtools: Add a helper for running Python tests
>
> Add the function samba_utils.RUN_PYTHON_TESTS for running a Python
> test. When building for multiple Python versions, all are tested.
>
> Also, add the list of configured Python interpreters to build config.
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> buildtools/wafsamba/samba_python.py | 6 ++++++
> buildtools/wafsamba/samba_utils.py | 16 ++++++++++++++++
> 2 files changed, 22 insertions(+)
>
> diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
> index a0e0ffa..8b20066 100644
> --- a/buildtools/wafsamba/samba_python.py
> +++ b/buildtools/wafsamba/samba_python.py
> @@ -9,6 +9,8 @@ from Configure import conf
> @conf
> def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
> # enable tool to build python extensions
> + interpreters = []
> +
> if conf.env['EXTRA_PYTHON']:
> conf.all_envs['extrapython'] = conf.env.copy()
> conf.setenv('extrapython')
> @@ -21,6 +23,7 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
> except Exception:
> warn('extra-python needs to be Python 3.3 or later')
> raise
> + interpreters.append(conf.env['PYTHON'])
> conf.setenv('default')
>
> conf.find_program('python', var='PYTHON', mandatory=mandatory)
> @@ -29,6 +32,9 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
> conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
> conf.check_python_version(version)
>
> + interpreters.append(conf.env['PYTHON'])
> + conf.env.python_interpreters = interpreters
> +
>
> @conf
> def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
> diff --git a/buildtools/wafsamba/samba_utils.py b/buildtools/wafsamba/samba_utils.py
> index e8bc0f3..540fe44 100644
> --- a/buildtools/wafsamba/samba_utils.py
> +++ b/buildtools/wafsamba/samba_utils.py
> @@ -386,6 +386,22 @@ def RUN_COMMAND(cmd,
> return -1
>
>
> +def RUN_PYTHON_TESTS(testfiles, pythonpath=None):
> + env = LOAD_ENVIRONMENT()
> + if pythonpath is None:
> + pythonpath = os.path.join(Utils.g_module.blddir, 'python')
> + result = 0
> + for interp in env.python_interpreters:
> + for testfile in testfiles:
> + cmd = "PYTHONPATH=%s %s %s" % (pythonpath, interp, testfile)
> + print('Running Python test with %s: %s' % (interp, testfile))
> + ret = RUN_COMMAND(cmd)
> + if ret:
> + print('Python test failed: %s' % cmd)
> + result = ret
> + return result
> +
> +
> # make sure we have md5. some systems don't have it
> try:
> from hashlib import md5
> --
> 2.1.0
>
>
> From d22b7758d07180e9755a07948751bf4c216c29ff Mon Sep 17 00:00:00 2001
> From: Petr Viktorin <pviktori at redhat.com>
> Date: Wed, 6 May 2015 18:05:18 +0200
> Subject: [PATCH 7/7] pytalloc: Test for all Python versions
>
> Signed-off-by: Petr Viktorin <pviktori at redhat.com>
> ---
> lib/talloc/wscript | 10 +---------
> 1 file changed, 1 insertion(+), 9 deletions(-)
>
> diff --git a/lib/talloc/wscript b/lib/talloc/wscript
> index c520294..8e61516 100644
> --- a/lib/talloc/wscript
> +++ b/lib/talloc/wscript
> @@ -148,18 +148,10 @@ def build(bld):
> def test(ctx):
> '''run talloc testsuite'''
> import Utils, samba_utils
> - env = samba_utils.LOAD_ENVIRONMENT()
> cmd = os.path.join(Utils.g_module.blddir, 'talloc_testsuite')
> ret = samba_utils.RUN_COMMAND(cmd)
> print("testsuite returned %d" % ret)
> - if 'USING_SYSTEM_PYTALLOC_UTIL' not in env.defines and not env.disable_python:
> - cmd = "PYTHONPATH=%s %s test_pytalloc.py" % (
> - os.path.join(Utils.g_module.blddir, 'python'),
> - env['PYTHON'],
> - )
> - pyret = samba_utils.RUN_COMMAND(cmd)
> - else:
> - pyret = 0
> + pyret = samba_utils.RUN_PYTHON_TESTS(['test_pytalloc.py'])
> print("python testsuite returned %d" % pyret)
> sys.exit(ret or pyret)
>
> --
> 2.1.0
>
More information about the samba-technical
mailing list