[Samba] tdbtool repack fails

[Andrej Gessel]

Hello,


as Andrew has already recognized, the idea behind is to reduce file size of database in filesystem. In my case this database needs to be mmaped on 2/2GB split 32bit arm machine, so I don't have address space for it, but I could have space, if file size could be equal to data size.

Thanks Andrew to show me the offline workaround.


After tdbbackup I got this information, but I did not start my DC yet.

tdb> info
Size of file/data: 1350275072/1050098055
Header offset/logical size: 0/1350275072
Number of records: 669737
Incompatible hash: no
Active/supported feature flags: 0x00000000/0x00000001
Robust mutexes locking: no
Smallest/average/largest keys: 12/57/242
Smallest/average/largest data: 72/1510/1235987
Smallest/average/largest padding: 25/397/309007
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 25
Smallest/average/largest free records: 12/725751/18142352
Number of hash chains: 10000
Smallest/average/largest hash chains: 26/66/116
Number of uncoalesced records: 0
Smallest/average/largest uncoalesced runs: 0/0/0
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes: 3/75/20/1/0/1/0

File size is as expected after repack.

For tdbbackup I need to take my DC offline, how about repack? If my database would be smaller, and repack could work, can I run it online, or should I take my DC offline as well?

Andrej


________________________________
Von: Andrew Bartlett <abartlet at samba.org>
Gesendet: Donnerstag, 2. August 2018 20:14
An: Andrej Gessel; 'samba'
Betreff: Re: [Samba] tdbtool repack fails

On Thu, 2018-08-02 at 15:53 +0200, Andrej Gessel via samba wrote:
> Hello list,
>
> I try to repack(tdbtool repack) my "dc=domain,dc=com.ldb" file:
>
> tdb> info
> Size of file/data: 3388084000/1050098055
> Header offset/logical size: 0/3388084000
> Number of records: 669737
> Incompatible hash: no
> Active/supported feature flags: 0x00000000/0x00000001
> Robust mutexes locking: no
> Smallest/average/largest keys: 12/57/242
> Smallest/average/largest data: 72/1510/1235987
> Smallest/average/largest padding: 5/412/252927
> Number of dead records: 2
> Smallest/average/largest dead records: 399912712/661866360/923820008
> Number of free records: 38999
> Smallest/average/largest free records: 12/18485/715170720
> Number of hash chains: 10000
> Smallest/average/largest hash chains: 26/66/116
> Number of uncoalesced records: 253
> Smallest/average/largest uncoalesced runs: 1/1/2
> Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
> 1/30/8/21/39/1/0
>
>
> it ends with this error:
>
> tdb_recovery_allocate: overflow recovery area
> tdb_transaction_prepare_commit: failed to setup recovery data
> ../lib/tdb/common/tdb.c:1055 Failed to commit
>
> I follow the code and found this line:
>
> transaction.c:783
>
>          /* New head will be at end of file. */
>          recovery_head = tdb->map_size;
>
> In my case it means following:
>
> my filesize is 3,3GB, old recovery_head was on 2064351232, size of old
> recovery was about 923820008, size of new recovery should be 1323794228,
> after tdb_free of old recovery area and setting the new recovery head to
> end of file(tdb->map_size), it is no more place for new recovery area,
> so the next check returns "overflow recovery area".
>
>
> I'm now thinking about how it should be fixed? I thinking about to
> improve the checks and maybe search for last data entry and set new
> recovery head behind it?

In the short term:

Take the DC offline, and then run tdbbackup on each backend file, then
replace them one by one.  That will do the repack, but into a new file
rather than with a transaction.

In the medium term patches to improve this TDB behaviour would be most
welcome, with tests, so we can avoid folks getting stuck.  This will
help us bridge the gap to LMDB more safely.

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba