[TDB] Patches for file and memory usage growth issues

[simo]

On Tue, 2011-04-12 at 00:38 +0930, Rusty Russell wrote:
> I didn't implement tdb_repack; I hope we won't get as fragmented as
> TDB1 does (we do full coalescing).
> 
> Simo, here's my tdb_repack rewrite.  It's lightly tested, but I'd be
> interested to see how much it helps you.  It's a bit more complex than
> I'd hoped...

Hi Rusty,
unfortunately my tests doesn't show results to be as good as I hoped
for. Here is the tree I used:
http://git.samba.org/?p=idra/samba.git;a=shortlog;h=refs/heads/tdb-memsize

Tests take a while to run so I haven't done them all with the latest
patches, the first 2 tests below w/o your patches also still had my
original patch that didn't change the 25% allocation overhead, only the
x2/x100 change for records bigger than 100k.

With your patch, for some reason memory allocation didn't improve much,
I still got a peak of 3.9G without compression and 3.2G with compression
enabled (I have an additional patch in the packages I built for my
fedora machine that allows me to toggel compression using an environment
variable so that I don't have to recompile tests and library). This
compares to previous values of 3.7G and 2.8G with compression, so it
seem that it actually get worse, but again in the previous runs I didn't
have the heuristic to change to 10% over-allocation on expand after
100MiB so take these results with a grain of salt, I will repeat those
runs to get a better assesment of the difference with just your patch
on/off.

Your patches also seemed to make the tests much slower :/

Also notice how your patches are causing the tdb to nd up beeing even
bigger but about 200MiB (and note that in the non-rusty runs I had
heuristics to always increase the DB by 25%, while with your patches
applied I changed them to increase the size by only 10% once the TDB
grows past 100MiB).

Simo.


Preliminary results:

No compression: ~ 20 min:
 
$ tdbtool tests_sysdb.nocomp/sssd.ldb 
tdb> info
Size of file/data: 1350402048/665083532
Number of records: 76630
Smallest/average/largest keys: 12/48/65
Smallest/average/largest data: 43/8630/1289140
Smallest/average/largest padding: 20/1813/322353
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 1
Smallest/average/largest free records: 544455060/544455060/544455060
Number of hash chains: 10000
Smallest/average/largest hash chains: 0/7/39
Number of uncoalesced records: 0
Smallest/average/largest uncoalesced runs: 0/0/0
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
0/49/10/40/0/0/0
 
--------------------------------------------------------------------------------
With compression: ~ 27 min:
 
$ tdbtool tests_sysdb.comp/sssd.ldb 
tdb> info
Size of file/data: 816607232/356695483
Number of records: 77394
Smallest/average/largest keys: 12/48/65
Smallest/average/largest data: 43/4560/65775
Smallest/average/largest padding: 20/652/16464
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 24991
Smallest/average/largest free records: 40/16281/112705252
Number of hash chains: 10000
Smallest/average/largest hash chains: 0/7/39
Number of uncoalesced records: 20151
Smallest/average/largest uncoalesced runs: 1/4/12
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
0/43/6/50/0/0/0
 
--------------------------------------------------------------------------------
Rusty's patch no compression: ~ 30 min:
 
$ tdbtool tests_sysdb.rusty.nocomp/sssd.ldb 
tdb> info
Size of file/data: 1538629632/850286307
Number of records: 90658
Smallest/average/largest keys: 12/51/65
Smallest/average/largest data: 43/9327/1289140
Smallest/average/largest padding: 4/48387/4294967234
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 23742
Smallest/average/largest free records: 56/25009/283268176
Number of hash chains: 10000
Smallest/average/largest hash chains: 0/7/39
Number of uncoalesced records: 19139
Smallest/average/largest uncoalesced runs: 1/4/12
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
0/55/285/39/0/0/0
 
--------------------------------------------------------------------------------
Rusty's patch + compression: ~ 40 min:
 
$ tdbtool tests_sysdb.rusty.comp/sssd.ldb 
tdb> info
Size of file/data: 1173598208/501708330
Number of records: 90130
Smallest/average/largest keys: 12/50/65
Smallest/average/largest data: 43/5516/65776
Smallest/average/largest padding: 4/816/16462
Number of dead records: 0
Smallest/average/largest dead records: 0/0/0
Number of free records: 24981
Smallest/average/largest free records: 56/23838/284974024
Number of hash chains: 10000
Smallest/average/largest hash chains: 0/7/40
Number of uncoalesced records: 20140
Smallest/average/largest uncoalesced runs: 1/4/12
Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
0/42/6/51/0/0/0


-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>