[Samba] Multi domain controller environment Ubuntu 12.04, replication and DNS updates broken

[Chris Alavoine]

Hi all,

Am posting this again with a more helpful subject line...

My 5 DC production domain (4.1.7 Ubuntu 12.04) is in a bit of a state.

I attempted an upgrade from 4.1.5 to 4.1.7 which appeared to work, but now
we have replication errors and am unable to add any new DNS entries. I am
now certain that we've fallen foul of the DomainDnsZones DeletedObjects
problem that I've been reading about in various posts on the lists.

My DC=DOMAINDNSZONES,DC=EXAMPLE,DC=INTERNAL,DC=COM.ldb files are now
between 3 and 4GB on each of the DC's. Doing an ldapsearch ( ldbsearch -H
DC=DOMAINDNSZONES,DC=ESSENCE,DC=INTERNAL,DC=COM.ldb 'isDeleted=TRUE' dn )on
each DC returns a different number of objects ranging from 387000 down to
88000 on the FSMO DC. Almost all of these are stale isDeleted entries.

I have lowered the tombstoneLifetime setting as suggested by other posters
on the lists and this appears to be slowly (very slowly) lowering the
number of records within the ldb domaindnszones file, my hope is that they
will lower sufficiently so that I can join a new working 4.1.12 DC to
domain.

I am currently attempting a Bind migration on a test DC as this is toted as
a possible fix (any successes out there with this?).

A matter of note for the lists: When I originally provisioned my domain
(classic upgrade from Samba3) I created a new OU for Groups and moved all
groups into it, this is a mistake if you want to migrate to Bind as the
migration script needs CN=DnsAdmins to be in Users OU, if it isn't the
script errors. I moved DnsAdmins back to Users to get the script to
complete.

At present I'm holding the domain together with bits of string and sticky
tape - having to reboot one of my DC's every 30 mins just to keep things
ticking over.

I have tried many variations of joining a new DC to the domain but that has
failed, so my current plan is to create a test version of my FSMO DC using
BIND_DLZ (using a current snapshot of the FSMO DC) and get things to a
working state there, and then replace this on the production site and
re-join new DC's to rebuild things. Obviously, not best practice but I
can't think of any other way of getting things stable again.

I have tried manually editing the .ldb files but they are so inflated now
that any vim edits just time out and error.

Thanks,
Chris.

-- 
ACS (Alavoine Computer Services Ltd)
Chris Alavoine
mob +44 (0)7724 710 730
www.alavoinecs.co.uk
http://twitter.com/#!/alavoinecs
http://www.linkedin.com/pub/chris-alavoine/39/606/192