[Samba] [Patches] AD Database corruption after upgrade from <= 4.6 to 4.7 (bug #13228)

Tue Jan 23 13:12:08 UTC 2018

Am 23.01.2018 um 00:05 schrieb Achim Gottinger via samba:
>
>
> Am 22.01.2018 um 22:12 schrieb Ralph Böhme:
>> On Mon, Jan 22, 2018 at 05:24:44PM +0100, Achim Gottinger via samba 
>> wrote:
>>> Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via samba:
>>>> Also DO NOT repair the following errors with samba-tool dbcheck!
>>>> "Remove duplicate links in attribute"
>>>> and
>>>> "ERROR: orphaned backlink"
>>>> as this removes the ability to repair the database
>>>> in the next round of patches!
>>>>
>>> I had this error after upgrading from 4.7.3 to 4.7.4 and used 
>>> samba-tool
>>> dbcheck --clean to get rid of them.
>>> Replication is still working. What kind of unrepairable corruption 
>>> can i
>>> expect now?
>> see the bug report for details, this can eg cause loss of group 
>> memberships or
>> generally speaking loss of linked-attributes.
>>
>> The only remede is comparing all objects for differences in 
>> linked-attributes
>> and restore overwritten forward-links from now dangling backlinks.
>>
>> We're currently also working on an improvement to dbcheck so it can 
>> detect such
>> corruption and fix it, but this will only work if you did *not* run 
>> dbcheck
>> --fix on the affected database.
>>
>> -slow
>>
> Thank you for the infos!
>
> I took a look at my notes.
>
> I updates from 4.6.8 to 4.7.3 on 25.11.2017.
>
> Back then i found error like this all related to siteList before the 
> update.
>
> ERROR: no target object found for GUID component for siteList in 
> object CN=DEFAULTIPSITELINK,CN=IP,CN=Inter-Site 
> Transports,CN=Sites,CN=Configuration,DC=samba-list,DC=loc - 
> <GUID=d4f41749a1595a43871ab1d72f24fe6b>;<RMD_ADDTIME=130015150890000000>;<RMD_CHANGETIME=130015150890000000>;<RMD_FLAGS=0>;<RMD_INVOCID=af301252bb781543b57dbd7cb773d46f>;<RMD_LOCAL_USN=4762>;<RMD_ORIGINATING_USN=4762>;<RMD_VERSION=0>;CN=Test,CN=Sites,CN=Configuration,DC=samba-list,DC=loc
> Not removing dangling forward link
> ERROR: no target object found for GUID component for siteList in 
> object CN=DEFAULTIPSITELINK,CN=IP,CN=Inter-Site 
> Transports,CN=Sites,CN=Configuration,DC=samba-list,DC=loc - 
> <GUID=596bd8ae9e8bc94eab99ad3c12e22132>;<RMD_ADDTIME=130739077850000000>;<RMD_CHANGETIME=130739077850000000>;<RMD_FLAGS=0>;<RMD_INVOCID=af301252bb781543b57dbd7cb773d46f>;<RMD_LOCAL_USN=453494>;<RMD_ORIGINATING_USN=453494>;<RMD_VERSION=0>;CN=Grafing,CN=Sites,CN=Configuration,DC=samba-list,DC=loc
> Not removing dangling forward link
> Please use --fix to fix these errors
>
> I updated to 4.7.3 and back then edited the ldb file and deleted the 
> links to old expunged sites whom did no longer exist with the given GUID.
>
> #~ldbedit -e nano -H 
> /varLib/samba/private/sam.ldb.d/CN=CONFIGURATION,DC=SAMBA-LIST,DC=LOC.ldb
> #~samba-tool dbcheck --reindexdb
>
> An month later on 26.12.2017 at about 5 am a few groups suddenly had 
> an messed up member list, some users showed up twice some where missing.
> I fixed it by deleting and recreating the affected groups, erros where 
> deceted but could not be fixed with samba-tool dbcheck for the 
> affected users/groups.
> Also deleting those twice listed users did not work. Thought it was 
> caused by an forced kill -9 to the samba service from an cron job at 
> that time.
>
> I maintain two separate networks with samba addc's and this only 
> happend at one of these networks, both run samba adds's on 5 and 7 
> sites. My thombstoneLifetime is set to 30 days ab both networks.
>
> On 12.01.2018 i updated from 4.7.3 to 4.7.4. dbcheck ran clean before 
> the update but showed a few dangling forward errors whom i then fixed 
> with dbcheck --fix. Till now no group corruption had happened.
> I can think of restoring an backup from 11.01.2018 to an vm with 4.7.4 
> here to inspect the errors from dbcheck again and maybe recreate these 
> deleted links again. As far as i remember the errors where different 
> on the ad's of whom i run a dozend, so this may become complicated.
>
> I assume the errors caused by the 4.6.8->4.7.3 update happened 30 days 
> later and I fixed these by recreating the affected groups. But i'm 
> unsure if the fixes i ran after the 4.7.3->4.7.4 update may cause 
> another corruption on 11.02.2018. dbcheck --cross-ncs did not find any 
> errors before the update only afterwards. So the question is will the 
> fixing of the newly detected  errors (by dbcheck version 4.7.4) cause 
> issues or are these unrelated.
>
> Achim
Did a few tests to answer my own questions.

Restored an backup from 23.12.2017 to an VM. At this point only one 
Computer Group had been comprimised. I used the -kcc workaround to 
prevent an immediate tomstone expunge.

With samba 4.7.3 i get these results:

#~samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in 
CN=WIN7-G-ADMIN,CN=Computers,DC=domain,DC=loc for link member in CN=CG 
Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc for link member in CN=CG 
Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
Please use --fix to fix these errors
Checked 556 objects (2 errors)

The errors can not be fixed with --fix.

With 4.7.4 the errors look different

#~samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in 
CN=WIN7-G-ADMIN,CN=Computers,DC=domain,DC=loc for link member in CN=CG 
Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
WARNING: Link (back) mismatch for 'memberOf' (1) on 
'CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc' to 'member' (2) on 
'CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc'
ERROR: Duplicate link values for attribute 'member' in 'CN=CG Grafing 
Laden,CN=Computers,DC=domain,DC=loc'
Duplicate link 
'<GUID=2eb2053a-19b3-4f0e-beaf-7c64fe577855>;<RMD_ADDTIME=130755196240000000>;<RMD_CHANGETIME=130755196240000000>;<RMD_FLAGS=0>;<RMD_INVOCID=521230af-78bb-4315-b57d-bd7cb773d46f>;<RMD_LOCAL_USN=457188>;<RMD_ORIGINATING_USN=457188>;<RMD_VERSION=0>;<SID=S-1-5-21-1446910239-1605792192-310601177-9714>;CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc'
Correct   link 
'<GUID=2eb2053a-19b3-4f0e-beaf-7c64fe577855>;<RMD_ADDTIME=130755196240000000>;<RMD_CHANGETIME=130755196240000000>;<RMD_FLAGS=0>;<RMD_INVOCID=521230af-78bb-4315-b57d-bd7cb773d46f>;<RMD_LOCAL_USN=457188>;<RMD_ORIGINATING_USN=457188>;<RMD_VERSION=0>;<SID=S-1-5-21-1446910239-1605792192-310601177-9714>;CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc'
Not removing duplicate links in attribute 'member'
Please use --fix to fix these errors
Checked 556 objects (2 errors)

The i forced the tombstone expunge

#~samba-tool domain tombstones expunge

Afterwards a few more groups where compromised.

samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in 
CN=haar,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=haar,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fhe,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fhe,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=an,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=an,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=an,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=lr,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=lho,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=lho,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=poing,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=poing,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=marktschwaben,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=marktschwaben,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rs,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rs,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rr,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rr,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fs,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fs,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sw,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sw,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=gd,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=tib,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=tib,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=bf,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ke,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ke,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=tb,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=tb,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=mg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=mg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=hg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=hg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ag,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ag,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=jg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=jg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sf,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sf,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=schwabing,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ug,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen 
Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=ug,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=alg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=alg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc for link member in CN=CG 
Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rg,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=rg,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=reitz,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fh,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=fh,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sk,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=sk,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=lk,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=lk,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=grafing,CN=Users,DC=domain,DC=loc for link member in CN=DG 
Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in 
CN=grafing,CN=Users,DC=domain,DC=loc for link member in CN=Email 
Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
Please use --fix to fix these errors
Checked 556 objects (62 errors)

Back then i had to delete and recreate this groups to fix the issues.

With 4.7.4 and the patch "fix linked attribute corruption on databases 
with" running "samba-tool domain tombstones expunge" does not cause the 
corruption of the above groups.

Afterwards i tested an backup from 11.01.2018 (before i upgraded from 
4.7.3 to 4.7.4). (Un)fortunately i can not reproduce the dbcheck errors 
i had seen on the production system. As far as i remeber these where 
small site related issues and not caused by bug #13228. Also did another 
tombstone expunge which did not remove any object and

So i assume with the groups issues already fixed and the patch applied 
to 4.7.4 I'm save from future issues by this bug.

Thanks for the info's and the patch

Sincere,
Achim~