Trying to understand upgradeprovision

Fri Feb 15 03:02:36 MST 2013

On Fri, 2013-02-15 at 00:50 -0800, Matthieu Patou wrote:
> On 02/14/2013 11:49 PM, Andrew Bartlett wrote:
> > On Thu, 2013-02-14 at 23:27 -0800, Matthieu Patou wrote:
> >> On 02/13/2013 09:31 PM, Andrew Bartlett wrote:
> >>> Matthieu,
> >>>
> >>> I've been adding tests to help me understand upgradeprovision, what it
> >>> can do and what it can't do.
> >>>
> >>> The branch this mail is based on is here:
> >>> https://git.samba.org/abartlet/samba.git/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/samba-tool-and-acls
> >>>
> >>> The issue I have is that I can't find any clear description of what
> >>> --full does, and how we can safely use upgradeprovision to correct the
> >>> ACLs that were incorrect in our 4.0.0 release.
> >> Full will not only correct file location it will also generate a
> >> reference provision and add all the objects that are in the reference
> >> provision but not in the live one, it will also look at missing and
> >> changed attributes and add them (well not all because some shouldn't be
> >> copied but you get the idea).
> >>> Is it the case that without --full, we only do things so clear that we
> >>> should probably do them in dbcheck instead (and have upgradeprovision
> >>> run dbcheck at the end?)
> >> Well dbcheck should check the ... not a kitchen-sink
> > Can you clarify for me *exactly* what upgradeprovision does in each
> > mode?  That is, can you expand on the above to spell out what a 'not
> > full' mode does, in more detail than 'correct file locations'?
> The guy who asked me about the non full mode is you so you should know 
> as good as me ...
> 
> There is also a whole documentation paragraph before the main function, 
> did you had a look at it ?

I have.  And despite that I find this some of the most incompressible
code in Samba.  I know you know it very well, but I don't.  That in turn
is why I come across as un-trusting of it - because not only do I find
it hard to understand what exactly the tool does, I also don't have a
good series of tests I can look at either. 

> Looking at the code the main difference between full and the rest is 
> that the full will do the update of the sam.ldb the other mode won't but 
> in both case we look for the SDs that needs to be updated (those which 
> are different from the ref database) and we try to update it.

OK.  What else does the default mode of operation do (as an overview)?

> > In terms of things I of what I would like to see dbcheck handle (and
> > upgradprovison call into dbcheck for), it is things where a specific
> > rule can be applied, such as "there must be a deleted objects
> > container", "links must have a backlink" etc.  For these, we don't need
> > the reference provision nor the provision USN range, as these rules
> > always apply.
> True, you still have to remember the history of why some stuff are 
> present here as we had a bumpy road toward a AD like provision.

I agree, and I'm glad we have this tool, because it made many important
things possible, particularly for our valuable early testing sites.  

> >>> I had been under the impression that the SD recalculation was done
> >>> regardless, but when I run these commands:
> >>>
> >>> bin/samba-tool ldapcmp st/provision/alpha13_upgrade/private/sam.ldb
> >>> st/provision/alpha13_upgrade_reference/private/sam.ldb --two domain
> >>> schema --sd > /tmp/ldapcmp-upgrade_sd.txt
> >>>
> >>> bin/samba-tool ldapcmp st/provision/alpha13_upgrade_full/private/sam.ldb
> >>> st/provision/alpha13_upgrade_reference/private/sam.ldb --two domain
> >>> schema --sd > /tmp/ldapcmp-upgrade_full_sd.txt
> >>>
> >>> I see that the SDs are only fixed if we run with --full.  My concern is
> >>> that this mode changes many, many other things about the database, and I
> >>> would prefer to find a 'lighter touch' way to handle this.
> >> Well the lighter way might be --fixdsacl
> > I don't see a --fixdsacl at the moment, so I presume you mean you would
> > patch up a new main mode, so we would have: the default mode, --fixdsacl
> > and --full?
> Why not, but looking more closely, it seems that for alpha13 at least 
> --fixdsacl won't be sufficient as without full with don't recalculate 
> them very correctly I don't know why but there must be something that 
> influence greatly how we do the calculation.

The tests I did to understand the behaviour here were against alpha13,
but the version I'm most concerned about upgrading from is 4.0.0 (and
versions prior to that with identical/similar templates).  If we expect
that the behaviour of the two versions will be quite different, then
I'll add another expanded tree to the repository. 

> The initial idea is that --full does *everything* that is needed, no 
> options was more lightweight, you add also a bunch of options for the 
> ntacls that maybe should be removed as they seems not very clear and we 
> have a script to do it now separately, so adding --fixdsacl whould be 
> for: do nothing but fixing the acls in the DS part.

This would be very valuable, and perhaps we might make that a dbcheck
option?  

For clarity (and in conjunction with the question I ask above), you you
spell out what would the difference then be between this and the
current, default upgradeprovision behaviour?

> >>> Anyway, I include the output from those commands, in the hope that you
> >>> can shed some light.  To reproduce, on my branch first run 'make test
> >>> TESTS=alpha13' and the directories will be produced.
> >> Well the results for the --full looks pretty good, just one object is
> >> not ok (maybe a bug ?)
> >>
> >> Comparing:
> >> 'CN=ARES,OU=Domain Controllers,DC=alpha13,DC=samba,DC=corp'
> >> [st/provision/alpha13_upgrade_full/private/sam.ldb]
> >> 'CN=ARES,OU=Domain Controllers,DC=alpha13,DC=samba,DC=corp'
> >> [st/provision/alpha13_upgrade_reference/private/sam.ldb]
> >>       ACEs found only in st/provision/alpha13_upgrade_full/private/sam.ldb:
> >>           (OA;;SW;DNS-Host-Name-Attributes;;DA)
> >>           (OA;;SW;DNS-Host-Name-Attributes;;PS)
> >>       ACEs found only in
> >> st/provision/alpha13_upgrade_reference/private/sam.ldb:
> >>           (OA;;SW;Validated-DNS-Host-Name;;DA)
> >>           (OA;;SW;Validated-DNS-Host-Name;;PS)
> >>       FAILED
> > Yes, the --full mode does very well, and if you re-run the tests without
> > the --sd argument, you will see that there are only a few objects that
> > are incorrectly upgraded from alpha13.  (in the test, we filter out the
> > attributes so I could have a test that passed for all other objects).
> Well It means ihmo that we have a bug in the recalculation function as 
> if we ask the function to recalculate the SD it should do it.

I know you are busy, but are you able to look into it and fix it?

> > I am looking for something that is a lighter touch however - not
> > potentially rewriting any attribute and any object, when we only changed
> > around 10 items in the provision template.  Do you think you could add
> > and test such a mode?
> I'm sorry I don't understand this phrase.

As you proposed with the --fixdsacl (I think), I'm looking for a mode of
operation that only changes security descriptors that we know changed
since 4.0.0.  I want this restricted mode of operation because we know
the changes to the default provision templates are restricted to these
10 or so attribute values, and so the command we propose to run should
not alter any other objects or attributes. 

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org