[PATCHES] freelist defragmentation and (dependent) ctdb repacking

[Michael Adam]

Any more comments?

For Amitay's concern, there are two (gradual) options
(if the changes are generally found to be worthwile):

1. don't publish freelist_merge_adjacent at all and only
   have freelist_size call it.
   Hence no tdb version bump and no tdbtool command "merge"
   (unless we use tdb_freelist_size)
2. publish freelist_merge_adjacent, and hence bump the tdb
   version number but also have freelist_size call
   merge_adjacent, so that we can have ctdb benefit
   from the defragmentation without requiring a newer
   version of tdb.

Cheers - Michael

On 2014-06-13 at 09:30 +0200, Michael Adam wrote:
> On 2014-06-13 at 17:10 +1000, Amitay Isaacs wrote:
> > >
> > 
> > Hi Michael,
> > 
> > The changes look good, but haven't finished reviewing them yet.
> 
> Thanks so far!
> 
> > One of the issue is that the ctdb changes are not compatible with older tdb
> > version.  Are we making a decision to drop the compatibility with older
> > versions of tdb?
> 
> Well I am not suggesting any such decision here.
> Therefore, we can omit the final patch to ctdb
> and simply have ctdb call tdb_freelist_size: That is
> the first approach, where tdb_freelist_size is
> changed to call tdb_freelist_merge_adjacend()
> so that the API does not change, but a defragmenting
> side-effect is added.
> 
> That's why I explicitly put it this way:
> 
> > On Thu, Jun 12, 2014 at 1:43 AM, Michael Adam <obnox at samba.org> wrote:
> > >
> > > - Because we might not want to bump the version to 1.3.1 for
> > >   this (not sure..), or as a proposal, I have also changed
> > >   tdb_freelist_size() to call tdb_freelist_merge_adjacent()
> > >   so that it automatically defragments.
> > >
> > > - Secondly, there is a place when we traverse the freelist
> > >   anyways, namely in tdb_allocate_from_freelist(). I have
> > >   changed our loop there to merge with left freelist records,
> > >   thereby automatically reducing the freelist fragmentation
> > >   as the database is used. This will usually not traverse until
> > >   the end though since the bes fit algorithm works with decreasing
> > >   accuracy.
> > >
> > > - I probably owe a test to measure the effect?!
> > >
> > > The second patchset is for ctdb and builds upon the first one.
> > >
> > > - It changes the vacuum code to always call tdb_freelist_size()
> > >   again before checking whether a repack run is needed, so that
> > >   it automatically defragments the freelist.
> > >   This might reduce the frequency of (blocking) repacks.
> > >
> > > - As a variant, on top there is a patch to explicitly call
> > >   tdb_freelist_merge_adjacent() instead of freelist_size().
> > >   If we choose to have freelist_size defragment, we don't
> > >   need this change, and it is also more backward compatible.
> 
> Is that the more reasonable approach?
> 
> Cheers - Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140616/49fb129f/attachment.pgp>