[patches] make upgradeprovision use replPropertyMetaData and others cools things

[Matthieu Patou]

Jelmer, Metze,

I pushed on my public repo at 
http://repo.or.cz/w/Samba/ekacnet.git/shortlog/refs/heads/upgradeprovision 
updated patches.

I'm currently working on more unittest for upgradeprovision

Here is the function of upgradeprovision that are not tested

def define_what_to_log(opts):
def messageprovision(text):
def message(what, text):
def check_for_DNS(refprivate, private):
def populate_links(samdb, schemadn):
def populate_dnsyntax(samdb, schemadn):
def sanitychecks(samdb, names):
def print_provision_key_parameters(names):
def handle_special_case(att, delta, new, old, usn):
def dump_denied_change(dn, att, flagtxt, current, reference):
def handle_special_add(samdb, dn, names):
def check_dn_nottobecreated(hash, index, listdn):
def add_missing_object(ref_samdb, samdb, dn, names, basedn, hash, index):
def gen_dn_index_hash(listMissing):
def add_deletedobj_containers(ref_samdb, samdb, names):
def add_missing_entries(ref_samdb, samdb, names, basedn, list):
def handle_links(samdb, att, basedn, dn, value, ref_value, delta):
def update_present(ref_samdb, samdb, basedn, listPresent, usns, 
invocationid):
def update_partition(ref_samdb, samdb, basedn, names, schema, 
provisionUSNs):
def check_updated_sd(ref_sam, cur_sam, names):
def fix_partition_sd(samdb, names):
def rebuild_sd(samdb, names):
def removeProvisionUSN(samdb):
def delta_update_basesamdb(refpaths, paths, creds, session, lp):
def simple_update_basesamdb(newpaths, paths, names):
def update_privilege(ref_private_path, cur_private_path):
def update_samdb(ref_samdb, samdb, names, highestUSN, schema):
def update_machine_account_password(samdb, secrets_ldb, names):
def update_gpo(paths, creds, session, names):
def getOEMInfo(samdb, rootdn):
def updateOEMInfo(samdb, names):
def setup_path(file):

Not all of them are meaningful to test (ie. message() or 
print_provision_key_parameters()).

For most of the rest, it's not testable in the current state but I plan 
to split the function so that the real database modification is done in 
a separate and simple function and the core logic is in another function 
that can be tested because we can feed it with test data.

Let me know if you have any remarks.

Matthieu.

On 25/04/2010 01:53, Matthieu Patou wrote:
> On 24/04/2010 16:05, Stefan (metze) Metzmacher wrote:
>> Hi Matthieu,
>>
>>> Please find attached 5 patches for provision and upgradeprovision
>>>
>>> 0001-s4-provision-Add-information-about-provisioned-usn-r.patch
>>> 0002-s4-upgrade-provision-Refactor-code-to-do-all-the-mod.patch
>>> 0003-s4-Add-functions-related-to-ldb-manipulation-when-do.patch
>>> 0004-s4-upgradeprovision-Inform-about-new-dns-dynamic-upd.patch
>>> 0005-s4-upgradeprovision-Use-replPropertyMetaData-for-bet.patch
>>>
>>>
>>> The first one is to add the @PROVISION object to the base sam.ldb, this
>>> object for the moment store the lastProvisionUSN attribute that 
>>> indicate
>>> which usn range has been touched by the provision. This information 
>>> will
>>> be used later on by the upgradeprovision to know which range can be
>>> safely modified and which can't ...
>>>
>>> The second one is a refactoring of the code of upgradeprovision to 
>>> allow
>>> to use 1 transaction so that it's easier to track usn modification
>>> during upgradeprovision and also if upgradeprovision fails it leaves 
>>> the
>>> database as it was before.
>>>
>>> The third patch is to add functions needed by upgradeprovision
>> Why are the registry functions commented out?
> Well this is because at the beginning I had the idea to have all the 
> ldb, but it turns out that path to registry ldbs is relative.
> And if I open those ldbs it will create the files in the current 
> directory.
> To my mind we should have non relative path but mathias seems to think 
> the opposite (and indeed at the present time having absolute path for 
> registry files will break things), so the function should be more 
> clever and add prefix when the path is not absolute.
> For the moment I don't feel it as a huge need, but I leaved the 
> comments so that it will allow me to remember to do it eventually.
>
> Maybe the comments should be removed ?
>>> The forth one is to inform about the new dns changes and the fact that
>>> maybe the sysadmin should do something.
>>>
>>> The last one is the introduction of replPropertyMetaData to drive the
>>> upgrade, the idea here is to modify only the current host is the owner
>>> of the attribute (originate_invocationid) and if the attribute was
>>> modified during a provision or a previous upgradeprovision. This
>>> information will only be available on new provision but there is a
>>> script that I have (and that I can share but in the same time I didn't
>>> wan't to spend ages on the cleaning) that creates an ldif file ready to
>>> be integrated in the current provision to populate the range (it use an
>>> educated guess to do so).
>> Why do we have special cases for "member"?
>>
> Well, for the pre-repl mode, it's because the upgrade tries to modify 
> as less as possible but when upgrading different version it turns out 
> that updating the "member" was a need, otherwise in the function 
> handle_special_case when we have a usn it's just a left over as links 
> are managed a bit before, I'll correct this dead code. Normally all 
> the link attribute are handled in handle_links.
>> Do we have some documentation that describes the logical steps
>> do do on the upgrade? At least for me it's hard to find out how this is
>> supposed to work.
> Well the code it self is documented but I can add some docs either in 
> the program itself or in a separate file.
>> I'm not sure it's correct to only "fix" local generated values.
> I'm not sure I understand what do you mean.
> Do you mean that maybe it's a better idea to make a new provision and 
> then force a replication from the old to the new provision ?
>> It might be better to generate a new invocationId for the database
>> (as the restore on from backup on windows does) and important the old
>> stuff as replicated updates.
> Well a few reason not to do so :
>
> * I suppose that the MS model is quite stable, for the moment it's 
> still not the case for samba provision (although we have less changes 
> lately)
> * I suppose you can't restore a DC when there is already one running 
> DC for the domain (or put it in another way: restore is a way to 
> recreate a domain and if you still have a DC then it's pointless to do 
> so as you just need to make a dcpromo to add a new DC), here we should 
> be able to run upgrade provision with a provision with more than 1 DC 
> (it was the sense of the modification that tridge suggested)
> * I'm hopping to have time to focus on openldap backend soon, for them 
> the replication through drs was (is?) not an option
> * The goal is also to be able to upgrade from older provision where 
> all this stuff is not necessarily available
> * If you change the invocationID it's more difficult after to 
> understand which attribute/object you can override because it has been 
> created/modified by (upgrade)provision as the rule for the core update 
> is allow a modification if the usn has been modified during 
> provision/upgradeprovision and if the current DC is the owner of this 
> modification.
>
> Matthieu
>> metze
>>
>