Generating loadparm from our XML documentation

[Andrew Bartlett]

On Thu, 2014-01-09 at 08:28 +0100, Volker Lendecke wrote:
> On Thu, Jan 09, 2014 at 12:24:14PM +1300, Andrew Bartlett wrote:
> > On Wed, 2014-01-08 at 11:13 +0100, Volker Lendecke wrote:
> > > On Tue, Jan 07, 2014 at 04:34:41PM +1300, Garming Sam wrote:
> > > > I've just put all the changes into a repository as the patch was too
> > > > big for the list.
> > > > 
> > > > git://git.catalyst.net.nz/samba.git rename-param
> > > 
> > > I agree that the names do not follow Samba's variable naming
> > > convention, but just doing a sweeping change is questionable
> > > at best: It will cause pain for people who have to port back
> > > and forth patches for no good functional reason, they don't
> > > even clean up convoluted code.
> > 
> > > Can you please explain why you need to rename all the
> > > parameters?
> > 
> > The future step of auto-generation requires this, as otherwise each of
> > these variable names would need to be encoded into our XML
> > documentation, which would increase the risk of errors being introduced
> > at that step.  While not perfectly consistent, the current _-separated
> > names generally match the parameter string. 
> 
> It does. However -- can we discuss how you envision the
> parameters to be handled in the future? The current
> situation is a mess at best, and before we go too far into
> one direction, I would love to see consensus how to do this
> in the future. You seem to indicate to generate code from
> XML? Wouldn't it be better to generate the XML from code?
> This would make XML handling optional for people who *just*
> want the code and don't care about manpages.

I'm very happy to discuss this, which is why I've tried to foreshadow
where I hope Garming's work might go since the first patches sent on
Christmas Eve. 

First, the fundamental idea (of generating this all from the docs) isn't
mine, or particularly new.  I'm pretty sure Michael Adam was the one to
suggest it to me during one of the previous re-factoring stages.  We
discussed it as a long-term goal in 
https://lists.samba.org/archive/samba-technical/2011-July/078525.html
and I also mentioned it in October when we fleshed out lib/param/README
with aca475b6bcbe364d09b83d17bf0504741ed3a84a.  Certainly I've had it
mentioned to me an number of times during the previous re-factoring that
while helpful, these steps have largely papered over problems rather
than really solved them. 

The new docs.py and any future generator is written in pure python,
using the builtin ElementTree, and by design requires no more of the
build host than the rest of our build system already does. 

As to if the XML documentation is the best source, I think it is.  While
XML is a highly frustrating language, it is structured, and we have
demonstrated the use of the python ElementTree API to read it and
extract the relevant elements, without disturbing the generation of the
manpages.   

Also, adding XML documentation isn't optional, because we have a test
(docs.py) that enforces that we have an entry in the smb.conf manpage
for every parameter.  The docs.py test is being extended to cover the
defaults, and that in turn has been behind many of the patches in the
other series posted here.  Once we cross-verify, writing a generator (so
far for param_functions.c) has shown itself to be entirely practical.  

The goal is to only record the details of a parameter once, and to have
a clean, clear generation system, first providing input to and then
replacing the current CPP and perl-based system.  As our per-parameter
smb.conf snippets already contain XML (to refer to other parameters, to
add emphasis etc), it seems best to keep them in XML files where editors
can check for conformance.  

Finally, the workflow for adding a parameter would that currently used
for documenting one, that is to add a new .xml file to
docs-xml/smbdotconf/misc with the details of the new parameter, using
existing parameters as an example.  

I hope this fleshes out this proposal a bit more. 

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba