[Samba] Repacking database from v1 to v2 format: how long does it take?

[Andrew Bartlett]

On Tue, 2019-09-24 at 11:44 +0200, Francesco Malvezzi via samba wrote:
> On 23/09/2019 20:41, Andrew Bartlett wrote:
> > On Mon, 2019-09-23 at 11:25 +0200, Francesco Malvezzi via samba
> > wrote:
> > > hi all,
> > > 
> > > I updated a small domain with 8k object to samba-4.11.0 and the
> > > database
> > > conversion from v1 to v2 didn't take a noticeable time.
> > > 
> > > On the other hand, in a larger domain with 67k object, where the
> > > sudo ./bin/samba-tool dbcheck --cross-ncs --fix
> > > takes ~40 minutes, 2 hours and half were not enough to complete
> > > the
> > > conversion.
> > > 
> > > Is it a couple of hours something expected if dbcheck takes so
> > > much
> > > time?
> > > 
> > > Is it maybe my AD db to be pathologic?
> > 
> > Ouch!  We did not see such long runtimes during our testing, which
> > is
> > why no major warnings were made.  Are you on TDB or LMDB?  Does the
> > host have enough RAM?
> 
> It's TDB. I am unaware of the chance to convert to LMDB.

Thanks for confirming that.  Currently the only way to convert is to
join a new DC using LMDB. 

We certainly could write a conversion tool, and perhaps we should, but
for now that is the only process.

> > 
> > At your scale I would suggest say 10G, being 4GB for the main TDB,
> > 4GB
> > for the transaction copy for the repack, and extra.
> 
> It's close: 8GB, but while doing repack memory's usage was never
> higher
> than 20%; it was high on cpu, one core out of two was stuck at 100%.
> According to top, there was no iowait, so I supposed it was not a
> memory
> issue.

Thanks for confirming that.  I think you are probably hitting some
pathological behaviours at the TDB layer when handling so many records,
perhaps in the free list.

The way to debug this, if you are keen to, is to do a debug build of
Samba run get a flame graph per:

http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions

That will get us a better idea of which routines are 'hot' during your
in-place upgrade.

I do wish you the very best with this and am keen to ensure this works
really well for you.

Andrew Bartlett
-- 
Andrew Bartlett                       https://samba.org/~abartlet/
Authentication Developer, Samba Team  https://samba.org
Samba Developer, Catalyst IT          
https://catalyst.net.nz/services/samba