[TDB] Patches for file and memory usage growth issues

simo idra at samba.org
Thu Apr 14 12:14:32 MDT 2011


On Wed, 2011-04-13 at 14:46 +0930, Rusty Russell wrote:
> On Wed, 13 Apr 2011 11:22:33 +0930, Rusty Russell <rusty at samba.org> wrote:
> > I'll test that now and give you a fresh tree, sorry!
> 
> OK, this time for sure!
> 
> http://git.samba.org/?p=rusty/samba.git;a=shortlog;h=refs/heads/tdb-repack
> 
> Now, the final patch might be a bad idea, so I've separated it out.
> It's probably what's making things slow, so I'd test without it first.
> 
> I haven't merged your expand size logic patch yet, so feel free to
> push that to master whenever you want.
> 
> The following changes since commit af45636166c7a0cb87630105d18ce489e7391525:
> 
>   Fix bug 8072 - PANIC: create_file_acl_common frees handle two times. (2011-04-09 02:05:15 +0200)
> 
> are available in the git repository at:
>   git://git.samba.org/rusty/samba.git tdb-repack
> 
> Rusty Russell (4):
>       tdb: fix transaction recovery area for converted tdbs.
>       tdb: tdb_repack() only when it's worthwhile.
>       tdb: tdb_repack in place.
>       tdb: strip unused padding when repacking.
> 
>  lib/tdb/common/freelist.c    |   14 ++--
>  lib/tdb/common/repack.c      |  215 ++++++++++++++++++++++++++++++++++++++++++
>  lib/tdb/common/tdb.c         |   91 ------------------
>  lib/tdb/common/tdb_private.h |    6 +
>  lib/tdb/common/transaction.c |   87 +++++++++++++-----
>  lib/tdb/libtdb.m4            |    2 +-
>  lib/tdb/wscript              |    2 +-
>  7 files changed, 294 insertions(+), 123 deletions(-)
>  create mode 100644 lib/tdb/common/repack.c


Hi Rusty,
unfortunately this still doesn't seem to really help.

I've slightly modified my tests so I've rerun 3 tests:
1. plain
2. with your first 3 patches
3. with all 4 patches

plain is the baseline and it includes my patches to use better
heuristics as published in my git tree.

With your patches I actually see both an increase in time spent and size
of the memory footprint as well as final size of the tdb unfortunately.

The strict repack looks certainly overkill with thess tests, although
the basic repack patches do not hit too hard on time spent although the
final tdb is still 300MiB larger than without the patch.

Also the memory footprint with your patches is considerably higher
between 3.5 and 3.7 gigs vs 3.1 gigs of RES mem with plain.

The test creates 25000 users ans 250 groups each with 25000 users.
There is a transaction around each group creation operation, so ~ 250
transactions.

HTH,
Simo.


PLAIN tdb:
         
        16:20.35 elapsed
        96%CPU
         
        MAXMEM:
        23391 simo      20   0 4369m 3.1g 1.9g R 99.9 82.2  15:18.37
        sysdb-tests
         
        tdbtool info:
        Size of file/data: 1048993792/665083314
        Number of records: 76628
        Smallest/average/largest keys: 12/48/65
        Smallest/average/largest data: 43/8631/1289140
        Smallest/average/largest padding: 20/1283/322353
        Number of dead records: 0
        Smallest/average/largest dead records: 0/0/0
        Number of free records: 1
        Smallest/average/largest free records:
        283650420/283650420/283650420
        Number of hash chains: 10000
        Smallest/average/largest hash chains: 0/7/40
        Number of uncoalesced records: 0
        Smallest/average/largest uncoalesced runs: 0/0/0
        Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
        0/63/9/27/0/0/0
        -------------------------------------------------------------------------------
         
Rusty's TDB REPACK:
         
        17:14.07 elapsed
        94%CPU
         
        MAXMEM:
        15882 simo      20   0 5306m 3.5g 2.2g R 99.7 91.5  15:40.56
        sysdb-tests
         
        tdbtool info:
        Size of file/data: 1378013184/820552100
        Number of records: 89944
        Smallest/average/largest keys: 12/50/65
        Smallest/average/largest data: 43/9072/1289140
        Smallest/average/largest padding: 9/1847/322353
        Number of dead records: 0
        Smallest/average/largest dead records: 0/0/0
        Number of free records: 5
        Smallest/average/largest free records: 80/77826236/344662528
        Number of hash chains: 10000
        Smallest/average/largest hash chains: 0/7/39
        Number of uncoalesced records: 0
        Smallest/average/largest uncoalesced runs: 0/0/0
        Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
        0/59/12/28/0/0/0
        -------------------------------------------------------------------------------
         
With Rusty's additional strict repack patch:
         
        26:51.14 elapsed
        96%CPU
         
        MAXMEM:
        19875 simo      20   0 5228m 3.7g 2.4g R 99.5 95.4  25:35.84
        sysdb-tests
         
        tdbtool info:
        Size of file/data: 1350995968/807554698
        Number of records: 90209
        Smallest/average/largest keys: 12/50/65
        Smallest/average/largest data: 43/8901/1289140
        Smallest/average/largest padding: 4/656/322301
        Number of dead records: 0
        Smallest/average/largest dead records: 0/0/0
        Number of free records: 24893
        Smallest/average/largest free records: 1772/19339/161526364
        Number of hash chains: 10000
        Smallest/average/largest hash chains: 0/7/39
        Number of uncoalesced records: 20054
        Smallest/average/largest uncoalesced runs: 1/4/12
        Percentage keys/data/padding/free/dead/rechdrs&tailers/hashes:
        0/59/4/36/0/0/0



-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>



More information about the samba-technical mailing list