Shares broken / utilities broken -- GSS server Update(krb5)(1) Update failed:

[Joe Comeaux]

Hey Steve,

> Maybe
> samba-tool domain demote -UAdministrator

Thanks for the quick response. I've already tried to remove the domain with the command line utilities ( /usr/local/samba/bin/samba-tool domain demote -UAdministrator ).
It seems to start off fine, detects the parent domain controller, prompts for admin password. But it goes down hill after that.
Entering in a bad password fails with "Wrong username or password" ( expected behaviour )
Entering in the correct password fails with :
root at jcomeaux:/usr/local/samba/bin# ./samba-tool domain demote -UAdministrator
Using atlas.loc.dom.smb as partner server for the demotion
Password for [LOC\Administrator]:
Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for e3514235-4b06-11d1-ab04-00c04fc2dcd2 at ncacn_ip_tcp:atlas.loc.dom.smb[1024,seal] NT_STATUS_UNSUCCESSFUL
ERROR(<class 'samba.drs_utils.drsException'>): uncaught exception - drsException: DRS connection to atlas.loc.dom.smb failed: (-1073741823, 'Undetermined error')
  File "/usr/local/samba/lib/python2.6/site-packages/samba/netcmd/__init__.py", line 160, in _run
    return self.run(*args, **kwargs)
  File "/usr/local/samba/lib/python2.6/site-packages/samba/netcmd/domain.py", line 260, in run
    (drsuapiBind, drsuapi_handle, supportedExtensions) = drsuapi_connect(server, lp, creds)
  File "/usr/local/samba/lib/python2.6/site-packages/samba/drs_utils.py", line 54, in drsuapi_connect
    raise drsException("DRS connection to %s failed: %s" % (server, e))

I believe I may have run in to a bit of a bind, as I believe to fix my kerberos ticket / account problems the solution may be to run the upgrade provision and recreate / resync those credentials.
In order to run upgradeprovision, I need to remove the second domain controller, but in order to remove the domain controller, the kerberos tickets need to be working properly.

On a side note, the error I get when trying to "manually" remove masteredBy attributes on the domain object are met with
failed to modify DC=loc,DC=dom,DC=smb - objectclass_attrs: attribute 'masteredBy' on entry 'DC=loc,DC=dom,DC=smb' must not be modified directly, it is a linked attribute
I dont have any problems removing the computer object for the actual secondary domain controller, but even after removing the computer object, the upgrade provision script still fails with 

Found 2 domain controllers. For the moment upgradeprovision is not able to handle an upgrade on a domain with more than one DC. Please demote the other DC(s) before upgrading
Sanity checks for the upgrade have failed. Check the messages and correct the errors before rerunning upgradeprovision


I believe if I re-create the ticket keytab(?) it may fix my problem, but my understanding of kerberos is very minimal, much less sambas integration with kerberos.
output of `klist -t -k /usr/local/samba/private/secrets.keytab ` is :

Keytab name: WRFILE:/usr/local/samba/private/secrets.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 06/07/11 14:19:57 HOST/atlas at LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas.LOC.DOM.smb at LOC.DOM.SMB
   1 06/07/11 14:19:57 ATLAS$@LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas at LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas.LOC.DOM.smb at LOC.DOM.SMB
   1 06/07/11 14:19:57 ATLAS$@LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas at LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas.LOC.DOM.smb at LOC.DOM.SMB
   1 06/07/11 14:19:57 ATLAS$@LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas at LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas.LOC.DOM.smb at LOC.DOM.SMB
   1 06/07/11 14:19:57 ATLAS$@LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas at LOC.DOM.SMB
   1 06/07/11 14:19:57 HOST/atlas.LOC.DOM.smb at LOC.DOM.SMB
   1 06/07/11 14:19:57 ATLAS$@LOC.DOM.SMB

output of kinit ATLAS$ with incorrect password is :
kinit: Preauthentication failed while getting initial credentials

output of kinit ATLAS$ with correct password is :
kinit: Clients credentials have been revoked while getting initial credentials

( I set the password for ATLAS$ with samba-tool utility, which I believe may have started this whole debacle )

Thanks
-Joe