[PATCH] CTDB vacuuming updates

[Martin Schwenke]

On Fri, 7 Nov 2014 08:03:38 +1100, Amitay Isaacs <amitay at gmail.com>
wrote:

> On Fri, Nov 7, 2014 at 4:44 AM, Michael Adam <obnox at samba.org> wrote:
> 
> > Hi Amitay,
> >
> > the commit messages look promising...
> > ...but where are the patches? :-)
> >
> > Cheers - Michael
> >
> > On 2014-11-06 at 23:28 +1100, Amitay Isaacs wrote:
> > > Hi,
> > >
> > > Here are some CTDB vacuuming changes to improve performance.
> > >
> > > [PATCH 1/6] ctdb-vacuum: Track time for vacuuming in database
> > >
> > > Add vacuuming latency to database statistics.
> > >
> > > [PATCH 2/6] ctdb-vacuum: Stagger vacuuming child processes
> > >
> > > On a highly loaded system, forking multiple vacuuming child processes
> > > simultaneously can be expensive.  So delay vacuuming event till there are
> > > no active vacuuming child processes.
> > >
> > > [PATCH 3/6] ctdb-vacuum: Use non-blocking lock when traversing delete
> > > [PATCH 4/6] ctdb-vacuum: Use non-blocking lock when traversing delete
> > >
> > > These change tdb_chainlock() to tdb_chainlock_nonblock().  Vacuuming
> > needs
> > > more work to re-add skipped records back into scheduled delete list.
> > > However, these patches still prevent vacuuming getting in the way of ctdb
> > > daemon.
> > >
> > > [PATCH 5/6] ctdb-vacuum: Do not delete VACUUM MIGRATED records
> > >
> > > While processing records scheduled for deletion, any record for which the
> > > current node is not lmaster, is migrated to the lmaster node.  For this
> > > migration recovery deamon creates a call_request with
> > > CTDB_CALL_FLAG_VACUUM_MIGRATION flag.  When such a record is stored on
> > > lmaster, the record is marked with CTDB_REC_FLAG_VACUUM_MIGRATED.  such
> > > records must be stored locally and scheduled for deletion locally.
> > >
> > > [PATCH 6/6] ctdb-recoverd: Process all the records for vacuum fetch
> > >
> > > Recovery daemon sends migration call requests one after another only
> > after
> > > receiving reply.  This is very slow and can take longer than the
> > > VacuumInterval.  Any subsequent migration request sent to recovery daemon
> > > will be ignored if the previous run has not completed.
> > >
> > >
> > > Please review and push if ok.
> > >
> > > Amitay.
> >
> 
> 
> Oops. :-)
> 
> Here are the patches!

Reviewed-by: Martin Schwenke <martin at meltin.net>

peace & happiness,
martin