[Samba] tdbtool repack fails

[Andrew Bartlett]

On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba wrote:
> Hello list,
> 
> I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:
> 
> tdb> info
> Size of file/data: 3388084000/1050098055
> Header offset/logical size: 0/3388084000
> Number of records: 669737
> Incompatible hash: no
> Active/supported feature flags: 0x00000000/0x00000001
> Robust mutexes locking: no
> Smallest/average/largest keys: 12/57/242
> Smallest/average/largest data: 72/1510/1235987
> Smallest/average/largest padding: 5/412/252927
> Number of dead records: 2
> Smallest/average/largest dead records: 399912712/661866360/923820008
> Number of free records: 38999
> Smallest/average/largest free records: 12/18485/715170720
> Number of hash chains: 10000
> Smallest/average/largest hash chains: 26/66/116
> Number of uncoalesced records: 253
> Smallest/average/largest uncoalesced runs: 1/1/2
> Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 
> 1/30/8/21/39/1/0
> 
> 
> it ends with this error:
> 
> tdb_recovery_allocate: overflow recovery area
> tdb_transaction_prepare_commit: failed to setup recovery data
> ../lib/tdb/common/tdb.c:1055 Failed to commit
> 
> I follow the code and found this line:
> 
> transaction.c:783
> 
>          /* New head will be at end of file. */
>          recovery_head = tdb->map_size;
> 
> In my case it means following:
> 
> my filesize is 3,3GB, old recovery_head was on 2064351232, size of old 
> recovery was about 923820008, size of new recovery should be 1323794228, 
> after tdb_free of old recovery area and setting the new recovery head to 
> end of file(tdb->map_size), it is no more place for new recovery area, 
> so the next check returns "overflow recovery area".
> 
> 
> I'm now thinking about how it should be fixed? I thinking about to 
> improve the checks and maybe search for last data entry and set new 
> recovery head behind it?

In the short term:

Take the DC offline, and then run tdbbackup on each backend file, then
replace them one by one.  That will do the repack, but into a new file
rather than with a transaction. 

In the medium term patches to improve this TDB behaviour would be most
welcome, with tests, so we can avoid folks getting stuck.  This will
help us bridge the gap to LMDB more safely. 

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba