[SCM] CTDB repository - branch 1.3 updated - ctdb-1.9.1-402-g4c7d18a
Michael Adam
obnox at samba.org
Mon Mar 14 08:40:28 MDT 2011
The branch, 1.3 has been updated
via 4c7d18a5430c8178d85a84da48c20c5322046c88 (commit)
via 9d6c6f2445a0221c19a384809f9c373a6aae9606 (commit)
via ce70adb83e9df4c68f702b8bb832fcc9f3d14dd4 (commit)
via da0505b7a9bdcd44f4f07b0f9448bcc00a9647d3 (commit)
via 88410f052466234da81a913ff597966dfdd92ed8 (commit)
via 03b3fe74880317574c1a0d53be7519abb8482409 (commit)
via eeb50ec76a1e2b5d32fa141536856c58e8a48c0d (commit)
via 6e5a14ffd57be4842fd290170ace8781d4155ca6 (commit)
via 8b6d839fbcaeefb98240c52d4b4ea8c6b0f41c0d (commit)
via fedb54a72e6228147bf366a2efc18f91ce9dd4c8 (commit)
via 1cc2e859a9c9615e94f345a31dfcef1fe04b6ffe (commit)
via 4107ed6c5c1e10807ac1ee0720dda2cc41660a1c (commit)
via 4417cf00184cab61af11b4e550c16c67793ea038 (commit)
via a00dfaacd31bdf84b088eedda5f68ecd67543a64 (commit)
via 82332b53612250241b5dfd2436f78145dd291c87 (commit)
via 2c13b964763a7bb4d836878b72fe563925659b7e (commit)
via ff35eb22d4bc10ba6b543f677577c353f0226daf (commit)
via 10f737b3aaec6047c8f1caac8568c4ad10c04bf0 (commit)
via b8f818126b4b5b4e288e52f2a493b079be24ed8d (commit)
via 437ac7be514a07e1c8955958abef5bbc11a0ac88 (commit)
via ef2c1537eb8b210f1cc70f1f305e77add72547e5 (commit)
via fd13ac3657acff2205451ab160045fd8aa6b2bd3 (commit)
via 41e133ece7607b0acb953b0000aa2b132ac442b2 (commit)
via 36e35e7e1851246721b9f666cc20bddb25a6bea1 (commit)
via e872201485718af6c8b6c6ef6946938b5f51eef1 (commit)
via 238432747adc32056b7b8342eebe999170a20a88 (commit)
via 17bfa615bf494621764c6bfe79fe78e7556ca2c0 (commit)
via a5fbb30563dbfb3b0e25ffc3c370154e00a3881e (commit)
via 774685de7bae87a0365089efb8dfc43cb7d93282 (commit)
via 1e320502e748df372933181b3782087488d81f90 (commit)
via 729097e4fbbba5b877044ce5736ffb8e1cacc97f (commit)
via c3c3126f12e80b87a3d7e8aec69728db43c4636e (commit)
via 0da4c823c60dd63dd9edb275d779c8d1ac472566 (commit)
via d0e1667736f4755fab87c355899458cd5dc9e0bc (commit)
via 113a995f38a05e3c2debd5b7b69a1d11eb7ec23d (commit)
via 1a860608a47f86db6ad0efd731c27116eafa6dd2 (commit)
via 13c83ac65e4b92dbb6632cc6a46fa0992f5c8a7a (commit)
via 5db075725d46df89c285700fb11dfcada22c5ce0 (commit)
via 759643b524404f23108d1ae031a832052b8d4edc (commit)
via a3f7ff0357fc4c68311a4300086f24909f178d62 (commit)
via 5a3a98e6833b250e5025b3a32995af141af9181c (commit)
via 346101f5f7de32f668c5272f4e9425e7ed49451a (commit)
via fbe6307aea9b1f68b9b46721d271647487054e0c (commit)
via 58a5fce9959b1c2f7a9f81b961b1ab795bd5423d (commit)
via b1ee082396656aaadfa4b5b791a6c9c87a2eb851 (commit)
via 312fdab0905f6554c64d738c934d85fdd5428323 (commit)
via 3ede4d95fcd7faea7cc6839c02cbad1613394505 (commit)
via b03487d941bdbf187879a61c5fc3d732ef8c95aa (commit)
via 9c49b5b29da4a6c49a4b18f3f068f89361f4bf99 (commit)
via f95b9ff9481b4aa9d90cd07937b244d94b5af10c (commit)
via f67a69955e6170a90493545a6ec960db4ddcab22 (commit)
via 799af2162c9b24ff9c2b1c0bcb77735f10277a80 (commit)
via b72e2efd61244b2dd4cbae4480ec931e3d91d405 (commit)
via 4f9cb43b37a71b1fb246a65b9450dc48e9745d46 (commit)
via 37c720fe116d009cd95fe2abf9d0f1aca5a8a00c (commit)
via 7d5f22732095ded3bcf84492a59b7953221b2169 (commit)
via 6142f369171f6d2efe7c7bfbbdf513109c5d3b42 (commit)
via 02a00f70ad44aa7b069eadfd9abf3ffa9cfa9f78 (commit)
via c967a925cacd2b6ecba63b7c7b2d5bee885f57ab (commit)
via 7aaa81110aca817cb21c0acada90e8dc7dc237fe (commit)
via 10bf024474f30f9370d804ffad0319d34c58d433 (commit)
via 626e8a23c9f8f782dcc27751696b72f2a05cce64 (commit)
via 59d676ed07e2ebd38764f45eee7c67a73bf363ee (commit)
from 76ee5f604c9adaaa7392842cfa2753bccb071e0f (commit)
http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=1.3
- Log -----------------------------------------------------------------
commit 4c7d18a5430c8178d85a84da48c20c5322046c88
Author: Michael Adam <obnox at samba.org>
Date: Fri Mar 11 16:05:44 2011 +0100
vacuum: fix a comment typo
commit 9d6c6f2445a0221c19a384809f9c373a6aae9606
Author: Michael Adam <obnox at samba.org>
Date: Fri Mar 11 15:57:45 2011 +0100
vacuum: use insert_record_into_delete_queue in ctdb_local_schedule_for_deletion.
This is to take advantage of the hash collision handling and logging
also in ctdb_local_schedule_for_deletion.
commit ce70adb83e9df4c68f702b8bb832fcc9f3d14dd4
Author: Michael Adam <obnox at samba.org>
Date: Fri Mar 11 15:55:52 2011 +0100
vacuum: refactor insert_record_into_delete_queue out of ctdb_control_schedule_for_deletion
commit da0505b7a9bdcd44f4f07b0f9448bcc00a9647d3
Author: Michael Adam <obnox at samba.org>
Date: Fri Mar 11 14:57:15 2011 +0100
vacuum: raise a debug level from INFO to DEBUG
when overwriting an existing entry in the delete_queue.
commit 88410f052466234da81a913ff597966dfdd92ed8
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 16:32:23 2011 +0100
ctdb_ltdb_store_server: honour the AUTOMATIC record flag
Do not delete empty records that carry this flag but store
them and schedule them for deletetion. Do not store the flag
in the ltdb though, since this is internal only and should not
be visible to the client.
commit 03b3fe74880317574c1a0d53be7519abb8482409
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 16:30:52 2011 +0100
ltdb: add the CTDB_REC_FLAG_AUTOMATIC to the initial header in ctdb_ltdb_fetch()
Signals that this record was not created by a client level store.
commit eeb50ec76a1e2b5d32fa141536856c58e8a48c0d
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 16:27:42 2011 +0100
ctdb_private.h: add record flag CTDB_REC_FLAG_AUTOMATIC
This is a flag that shall signa that a record has been automatically generated by ctdb
and not by an explicit client store operation. This will be used in the ctdb_ltdb_fetch
operation which stores an empty record with default initial header before trying to
migrate the record from the dmaster when the record does not exist in the local tdb.
commit 6e5a14ffd57be4842fd290170ace8781d4155ca6
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 28 13:19:22 2010 +0100
ctdb_ltdb_store_server: add ability to send SCHEDULE_FOR_DELETION control to ctdb_ltdb_store.
commit 8b6d839fbcaeefb98240c52d4b4ea8c6b0f41c0d
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 18:08:11 2010 +0100
ctdb_ltdb_store_server: Improve debug message in ctdb_ltdb_store when store or delete fails.
commit fedb54a72e6228147bf366a2efc18f91ce9dd4c8
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 17:50:52 2010 +0100
ctdb_ltdb_store_server: always store the data when ctdb_ltdb_store() is called from the client
This also fixes a segfault since ctdb_lmaster uses the vnn_map.
commit 1cc2e859a9c9615e94f345a31dfcef1fe04b6ffe
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:13:50 2010 +0100
ctdb_ltdb_store_server: implement fastpath vacuuming deletion based on VACUUM_MIGRATED flag.
When the record has been obtained by the lmaster as part of the vacuuming-fetch
handler and it is empty and never been migrated with data, then such records
are deleted instead of being stored. These records have automatically been
deleted when leaving the former dmaster, so that they vanish for good when
hitting the lmaster in this way. This will reduces the load on traditional
vacuuming.
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
commit 4107ed6c5c1e10807ac1ee0720dda2cc41660a1c
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 3 15:29:21 2010 +0100
ctdb_ltdb_store_server: delete an empty record that is safe to delete instead of storing locally.
When storing a record that is being migrated off to another node
and has never been migrated with data, then we can safely delete it
from the local tdb instead of storing the record with empty data.
Note: This record is not deleted if we are its lmaster or dmaster.
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
commit 4417cf00184cab61af11b4e550c16c67793ea038
Author: Michael Adam <obnox at samba.org>
Date: Thu Dec 30 18:19:32 2010 +0100
server: Use the ctdb_ltdb_store_server() in the ctdb daemon for non-persistent dbs
This is realized by adding a ctdb_ltdb_store_fn function pointer to the db
context and filling it in the attach procedure for non-persistent dbs.
commit a00dfaacd31bdf84b088eedda5f68ecd67543a64
Author: Michael Adam <obnox at samba.org>
Date: Thu Dec 30 17:44:51 2010 +0100
server: create a server variant ctdb_ltdb_store_server() of ctdb_ltdb_store().
This is supposed to contain logic for deleting records that are safe
to delete and scheduling records for deletion. It will be called in
server context for non-persistent databases instead of the standard
ctdb_ltdb_store() function.
commit 82332b53612250241b5dfd2436f78145dd291c87
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 28 13:14:23 2010 +0100
daemon: fill ctdb->ctdbd_pid early
commit 2c13b964763a7bb4d836878b72fe563925659b7e
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 15:29:46 2010 +0100
test: send SCHEDULE_FOR_DELETION control from randrec test.
commit ff35eb22d4bc10ba6b543f677577c353f0226daf
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 15:29:23 2010 +0100
client: add accessor function ctdb_header_from_record_handle().
commit 10f737b3aaec6047c8f1caac8568c4ad10c04bf0
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 28 13:13:34 2010 +0100
vacuum: add ctdb_local_schedule_for_deletion()
commit b8f818126b4b5b4e288e52f2a493b079be24ed8d
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 14:25:48 2010 +0100
server: implement a new control SCHEDULE_FOR_DELETION to fill the delete_queue.
commit 437ac7be514a07e1c8955958abef5bbc11a0ac88
Author: Michael Adam <obnox at samba.org>
Date: Wed Mar 9 00:57:55 2011 +0100
control: add a new control opcode CTDB_CONTROL_SCHEDULE_FOR_DELETION
commit ef2c1537eb8b210f1cc70f1f305e77add72547e5
Author: Michael Adam <obnox at samba.org>
Date: Wed Mar 9 00:56:25 2011 +0100
control: add macro CHECK_CONTROL_MIN_DATA_SIZE.
This is for the control dispatcher to check whether the input data has
a required minimum size.
commit fd13ac3657acff2205451ab160045fd8aa6b2bd3
Author: Michael Adam <obnox at samba.org>
Date: Thu Dec 23 11:54:09 2010 +0100
vacuum: lower level of hash collision debug message to INFO
commit 41e133ece7607b0acb953b0000aa2b132ac442b2
Author: Michael Adam <obnox at samba.org>
Date: Thu Dec 23 00:27:27 2010 +0100
vacuum: add statistics output to the fast and full traverse runs.
commit 36e35e7e1851246721b9f666cc20bddb25a6bea1
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 14:19:00 2010 +0100
vacuum: refactor insert_delete_record_data_into_tree() out of add_record_to_delete_tree()
for reuse in filling the delete_queue.
commit e872201485718af6c8b6c6ef6946938b5f51eef1
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 21:43:41 2010 +0100
vacuum: change all Vacuum*Interval tunables to default to 10
So, by default we have a fastpath vacuuming every 10 seconds and
full blown db-traverse vacuuming once every 10 minutes.
commit 238432747adc32056b7b8342eebe999170a20a88
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 21:30:39 2010 +0100
vacuum: disable full db-traverse vacuuming runs when VacuumFastPathCount == 0
commit 17bfa615bf494621764c6bfe79fe78e7556ca2c0
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 18:03:38 2010 +0100
vacuum: Only run full vacuumig (db traverse) every VacuumFastPathCount times.
commit a5fbb30563dbfb3b0e25ffc3c370154e00a3881e
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:54:04 2010 +0100
vacuum: reset the fast path count in the event handle if it exceeds the limit.
commit 774685de7bae87a0365089efb8dfc43cb7d93282
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:49:29 2010 +0100
vacuum: bump the number of fast-path runs in the vacuum child destructor
commit 1e320502e748df372933181b3782087488d81f90
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:44:02 2010 +0100
vacuum: add a fast_path_count to the vacuum_handle.
commit 729097e4fbbba5b877044ce5736ffb8e1cacc97f
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:42:25 2010 +0100
Add a tunable VacuumFastPathCount.
This will control how many fast-path vacuuming runs wil have to
be done, before a full vacuuming will be triggered, i.e. one with
a db-traversal.
commit c3c3126f12e80b87a3d7e8aec69728db43c4636e
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:25:35 2010 +0100
vacuum: traverse the delete_queue befor traversing the database.
commit 0da4c823c60dd63dd9edb275d779c8d1ac472566
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:24:32 2010 +0100
vacuum: add delete_queue_traverse() for traversal of the delete_queue.
commit d0e1667736f4755fab87c355899458cd5dc9e0bc
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 11:22:50 2010 +0100
vacuum: reduce indentation in add_record_to_delete_tree()
This simplyfies the logical structure a bit by using early return.
commit 113a995f38a05e3c2debd5b7b69a1d11eb7ec23d
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 17:11:27 2010 +0100
vacuum: refactor new add_record_to_delete_tree() out of vacuum_traverse().
This will be reused by the traversal of the delete_queue list.
commit 1a860608a47f86db6ad0efd731c27116eafa6dd2
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 16:41:13 2010 +0100
vacuum: skip adding records to list of records to send to lmaster on lmaster
This list is skipped afterwards when the lists are processed.
commit 13c83ac65e4b92dbb6632cc6a46fa0992f5c8a7a
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 16:31:27 2010 +0100
vacuum: refactor new add_record_to_vacuum_fetch_list() out of vacuum_traverse().
This is the function that fills the list of records to send to each lmaster
with the VACUUM_FETCH message.
This function will be reused in the traverse function for the delete_queue.
commit 5db075725d46df89c285700fb11dfcada22c5ce0
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 10:55:53 2010 +0100
server: rename ctdb_repack_db() to ctdb_vacuum_and_repack_db()
commit 759643b524404f23108d1ae031a832052b8d4edc
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 17 02:22:02 2010 +0100
When wiping a database, clear the delete_queue.
commit a3f7ff0357fc4c68311a4300086f24909f178d62
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 17 01:53:25 2010 +0100
vaccum: clear the fast-path vacuuming delete_queue after creating the vacuuming child.
Maybe we should keep a copy for the case that the vacuuming fails?
commit 5a3a98e6833b250e5025b3a32995af141af9181c
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 17 01:38:09 2010 +0100
When attaching to a non-persistent DB, initialize the delete_queue.
commit 346101f5f7de32f668c5272f4e9425e7ed49451a
Author: Michael Adam <obnox at samba.org>
Date: Wed Dec 22 14:50:53 2010 +0100
Add a delete_queue to the ctdb database context struct.
This list will be filled by the client using a new
delete control. The list will then be used to implement
a fast-path vacuuming that will traverse this list instead
of traversing the database.
commit fbe6307aea9b1f68b9b46721d271647487054e0c
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:11:38 2010 +0100
call: becoming dmaster in VACUUM_MIGRATION, set the VACUUM_MIGRATED record flag
This temporary flag is used for the local record storage function to
decide whether to delete an empty record which has never been migrated
with data as part of the fast-path vacuuming process or, or to store
the record.
commit 58a5fce9959b1c2f7a9f81b961b1ab795bd5423d
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:07:21 2010 +0100
call: hand the submitted record_flags to local record storage function.
commit b1ee082396656aaadfa4b5b791a6c9c87a2eb851
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:02:33 2010 +0100
call: transfer the record flags in the ctdb call packets.
This way, the MIGRATED_WITH_DATA information can be transported
along with the records. This is important for vacuuming to function
properly.
The record flags are appended to the data section of the ctdb_req_dmaster
and ctdb_reply_dmaster structs.
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
commit 312fdab0905f6554c64d738c934d85fdd5428323
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 13:59:37 2010 +0100
server: in the VACUUM_FETCH handler, add the VACUUM_MIGRAION to the call flags
This way, the records coming in via this handler, can be treated appropriately.
Namely, they can be deleted instead of being stored when the meet the fast-path
vacuuming criteria (empty, never migrated with data...)
commit 3ede4d95fcd7faea7cc6839c02cbad1613394505
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 13:57:01 2010 +0100
add a new record flag CTDB_REC_FLAG_VACUUM_MIGRATED.
This is to be used internally. The purpose is to flag a record
as been migrated by a VACUUM_MIGRATION, which is triggered by
a VACUUM_FETCH message as part of the vacuuming. The local store
routine will base its decision whether to delete or to store
the record (among other things) upon the value of this flag.
This flag should never be stored in the local database copies.
commit b03487d941bdbf187879a61c5fc3d732ef8c95aa
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:22:55 2010 +0100
call: Move definition of call flags down to the definition of the flags field.
commit 9c49b5b29da4a6c49a4b18f3f068f89361f4bf99
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 10 14:24:40 2010 +0100
call: add new call flag CTDB_CALL_FLAG_VACUUM_MIGRATION
This is to be used when the CTDB_SRVID_VACUUM_FETCH message
triggers the migration of deleted records to the lmaster.
The lmaster can then delete records that have not been
migrated with data instead of storing them.
commit f95b9ff9481b4aa9d90cd07937b244d94b5af10c
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 3 15:24:06 2010 +0100
recoverd: in a recovery, set the MIGRATED_WITH_DATA flag on all records
Those records that are kept after recovery, are non-empty, and
stored identically on all nodes. So this is as if they had been
migrated with data.
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
commit f67a69955e6170a90493545a6ec960db4ddcab22
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 3 15:21:51 2010 +0100
server: when we migrate off a record with data, set the MIGRATED_WITH_DATA flag
commit 799af2162c9b24ff9c2b1c0bcb77735f10277a80
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 12:15:41 2011 +0100
vacuum: check lmaster against num_nodes instead of vnn_map->size
When lmaster is bigger than the biggest recorded node number,
then exit the traverse with error.
commit b72e2efd61244b2dd4cbae4480ec931e3d91d405
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 17:47:36 2011 +0100
vacuum: reduce indentation of the loop sending VACUUM_FETCH controls
This slightly improves the code structure in that loop.
commit 4f9cb43b37a71b1fb246a65b9450dc48e9745d46
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 12:26:45 2011 +0100
vacuum: correctly send TRY_DELETE_RECORDS ctrl to all active nodes
Originally, the control was sent to all records in the vnn_map, but
there was something still missing here:
When a node can not become lmaster (via CTDB_CAPABILITY_LMASTER=no)
then it will not be part of the vnn_map. So such a node would
be active but never receive the TRY_DELETE_RECORDS control from a
vacuuming run.
This is fixed in this change by correctly building the list of
active nodes first in the same way that the recovery process does it.
commit 37c720fe116d009cd95fe2abf9d0f1aca5a8a00c
Author: Michael Adam <obnox at samba.org>
Date: Thu Feb 3 12:18:58 2011 +0100
vacuum: in ctdb_vacuum_db, fix the length of the array of vacuum fetch lists
This patch fixes segfaults in the vacuum child when at least one
node has been stopped or removed from the cluster:
The size of the vnn_map is only the number of active nodes
(that can be lmaster). But the node numbers that are referenced
by the vnn_map spread over all configured nodes.
Since the array of vacuum fetch lists is referenced by the
key's lmaster's node number later on, the array needs to
be of size num_nodes instad of vnn_map->size.
commit 7d5f22732095ded3bcf84492a59b7953221b2169
Author: Michael Adam <obnox at samba.org>
Date: Mon Dec 20 16:26:50 2010 +0100
Fix typos in a comment in vacuum_traverse.
commit 6142f369171f6d2efe7c7bfbbdf513109c5d3b42
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 17:18:03 2010 +0100
tests: fix segfault in store test when connection to ctdbd failed.
commit 02a00f70ad44aa7b069eadfd9abf3ffa9cfa9f78
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 17:15:41 2010 +0100
tests: fix segfault in fetch_one test when connection to ctdbd fails
commit c967a925cacd2b6ecba63b7c7b2d5bee885f57ab
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 17:14:33 2010 +0100
tests: fix segfault in fetch test when connection to ctdb failed.
commit 7aaa81110aca817cb21c0acada90e8dc7dc237fe
Author: Michael Adam <obnox at samba.org>
Date: Tue Dec 21 17:11:26 2010 +0100
tests: fix segfault in randrec test when connection to daemon fails.
commit 10bf024474f30f9370d804ffad0319d34c58d433
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 3 15:39:44 2010 +0100
gitignore: add tags file
commit 626e8a23c9f8f782dcc27751696b72f2a05cce64
Author: Michael Adam <obnox at samba.org>
Date: Fri Dec 3 15:39:26 2010 +0100
gitignore: add vi swap files
commit 59d676ed07e2ebd38764f45eee7c67a73bf363ee
Author: Michael Adam <obnox at samba.org>
Date: Wed Nov 24 08:01:01 2010 +0100
server: add a comment explaining the call redirect logic in ctdb_call_send_redirect().
-----------------------------------------------------------------------
Summary of changes:
.gitignore | 2 +
client/ctdb_client.c | 9 +
common/ctdb_ltdb.c | 5 +
include/ctdb_private.h | 35 +++
include/ctdb_protocol.h | 6 +-
server/ctdb_call.c | 79 +++++-
server/ctdb_control.c | 9 +
server/ctdb_daemon.c | 1 +
server/ctdb_freeze.c | 11 +
server/ctdb_ltdb_server.c | 204 ++++++++++++++-
server/ctdb_recoverd.c | 2 +
server/ctdb_tunables.c | 7 +-
server/ctdb_vacuum.c | 667 +++++++++++++++++++++++++++++++++++++------
tests/src/ctdb_fetch.c | 5 +
tests/src/ctdb_fetch_one.c | 5 +
tests/src/ctdb_randrec.c | 48 +++-
tests/src/ctdb_store.c | 5 +
17 files changed, 991 insertions(+), 109 deletions(-)
Changeset truncated at 500 lines:
diff --git a/.gitignore b/.gitignore
index a4cc47e..063e1ed 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
*.[oa]
*~
+*.swp
config.status
configure
ctdb.pc
@@ -7,6 +8,7 @@ publish*.sh
push*.sh
web/packages
TAGS
+tags
bin
Makefile
config.h
diff --git a/client/ctdb_client.c b/client/ctdb_client.c
index 99ff72d..a43710f 100644
--- a/client/ctdb_client.c
+++ b/client/ctdb_client.c
@@ -4234,3 +4234,12 @@ int ctdb_ctrl_getstathistory(struct ctdb_context *ctdb, struct timeval timeout,
return 0;
}
+
+struct ctdb_ltdb_header *ctdb_header_from_record_handle(struct ctdb_record_handle *h)
+{
+ if (h == NULL) {
+ return NULL;
+ }
+
+ return &h->header;
+}
diff --git a/common/ctdb_ltdb.c b/common/ctdb_ltdb.c
index 200cca4..3ee7fe8 100644
--- a/common/ctdb_ltdb.c
+++ b/common/ctdb_ltdb.c
@@ -65,6 +65,7 @@ static void ltdb_initial_header(struct ctdb_db_context *ctdb_db,
ZERO_STRUCTP(header);
/* initial dmaster is the lmaster */
header->dmaster = ctdb_lmaster(ctdb_db->ctdb, &key);
+ header->flags = CTDB_REC_FLAG_AUTOMATIC;
}
@@ -129,6 +130,10 @@ int ctdb_ltdb_store(struct ctdb_db_context *ctdb_db, TDB_DATA key,
int ret;
bool seqnum_suppressed = false;
+ if (ctdb_db->ctdb_ltdb_store_fn) {
+ return ctdb_db->ctdb_ltdb_store_fn(ctdb_db, key, header, data);
+ }
+
if (ctdb->flags & CTDB_FLAG_TORTURE) {
struct ctdb_ltdb_header *h2;
rec = tdb_fetch(ctdb_db->ltdb->tdb, key);
diff --git a/include/ctdb_private.h b/include/ctdb_private.h
index 447d40c..5729e84 100644
--- a/include/ctdb_private.h
+++ b/include/ctdb_private.h
@@ -119,6 +119,7 @@ struct ctdb_tunable {
uint32_t allow_unhealthy_db_read;
uint32_t stat_history_interval;
uint32_t deferred_attach_timeout;
+ uint32_t vacuum_fast_path_count;
};
/*
@@ -514,6 +515,12 @@ struct ctdb_db_context {
struct lockwait_handle *lockwait_active;
struct lockwait_handle *lockwait_overflow;
struct ctdb_persistent_state *persistent_state;
+ struct trbt_tree *delete_queue;
+ int (*ctdb_ltdb_store_fn)(struct ctdb_db_context *ctdb_db,
+ TDB_DATA key,
+ struct ctdb_ltdb_header *header,
+ TDB_DATA data);
+
};
@@ -840,6 +847,14 @@ ctdb_control_send(struct ctdb_context *ctdb,
} \
} while (0)
+#define CHECK_CONTROL_MIN_DATA_SIZE(size) do { \
+ if (indata.dsize < size) { \
+ DEBUG(0,(__location__ " Invalid data size in opcode %u. Got %u expected >= %u\n", \
+ opcode, (unsigned)indata.dsize, (unsigned)size)); \
+ return -1; \
+ } \
+ } while (0)
+
int ctdb_control_getvnnmap(struct ctdb_context *ctdb, uint32_t opcode, TDB_DATA indata, TDB_DATA *outdata);
int ctdb_control_setvnnmap(struct ctdb_context *ctdb, uint32_t opcode, TDB_DATA indata, TDB_DATA *outdata);
int ctdb_control_getdbmap(struct ctdb_context *ctdb, uint32_t opcode, TDB_DATA indata, TDB_DATA *outdata);
@@ -1374,4 +1389,24 @@ int ctdb_deferred_drop_all_ips(struct ctdb_context *ctdb);
int ctdb_process_deferred_attach(struct ctdb_context *ctdb);
+/**
+ * structure to pass to a schedule_for_deletion_control
+ */
+struct ctdb_control_schedule_for_deletion {
+ uint32_t db_id;
+ struct ctdb_ltdb_header hdr;
+ uint32_t keylen;
+ uint8_t key[1]; /* key[] */
+};
+
+int32_t ctdb_control_schedule_for_deletion(struct ctdb_context *ctdb,
+ TDB_DATA indata);
+
+
+int32_t ctdb_local_schedule_for_deletion(struct ctdb_db_context *ctdb_db,
+ const struct ctdb_ltdb_header *hdr,
+ TDB_DATA key);
+
+struct ctdb_ltdb_header *ctdb_header_from_record_handle(struct ctdb_record_handle *h);
+
#endif
diff --git a/include/ctdb_protocol.h b/include/ctdb_protocol.h
index b6b753c..0422afe 100644
--- a/include/ctdb_protocol.h
+++ b/include/ctdb_protocol.h
@@ -34,13 +34,14 @@
#define CTDB_FETCH_FUNC 0xFF000002
-#define CTDB_IMMEDIATE_MIGRATION 0x00000001
struct ctdb_call {
int call_id;
TDB_DATA key;
TDB_DATA call_data;
TDB_DATA reply_data;
uint32_t status;
+#define CTDB_IMMEDIATE_MIGRATION 0x00000001
+#define CTDB_CALL_FLAG_VACUUM_MIGRATION 0x00000002
uint32_t flags;
};
@@ -361,6 +362,7 @@ enum ctdb_controls {CTDB_CONTROL_PROCESS_EXISTS = 0,
CTDB_CONTROL_SET_IFACE_LINK_STATE = 125,
CTDB_CONTROL_TCP_ADD_DELAYED_UPDATE = 126,
CTDB_CONTROL_GET_STAT_HISTORY = 127,
+ CTDB_CONTROL_SCHEDULE_FOR_DELETION = 128,
};
/*
@@ -482,6 +484,8 @@ struct ctdb_ltdb_header {
uint32_t reserved1;
#define CTDB_REC_FLAG_DEFAULT 0x00000000
#define CTDB_REC_FLAG_MIGRATED_WITH_DATA 0x00010000
+#define CTDB_REC_FLAG_VACUUM_MIGRATED 0x00020000
+#define CTDB_REC_FLAG_AUTOMATIC 0x00040000
uint32_t flags;
};
diff --git a/server/ctdb_call.c b/server/ctdb_call.c
index be6e8f9..73072c3 100644
--- a/server/ctdb_call.c
+++ b/server/ctdb_call.c
@@ -99,9 +99,30 @@ static void ctdb_send_error(struct ctdb_context *ctdb,
}
-/*
- send a redirect reply
-*/
+/**
+ * send a redirect reply
+ *
+ * The logic behind this function is this:
+ *
+ * A client wants to grab a record and sends a CTDB_REQ_CALL packet
+ * to its local ctdb (ctdb_request_call). If the node is not itself
+ * the record's DMASTER, it first redirects the packet to the
+ * record's LMASTER. The LMASTER then redirects the call packet to
+ * the current DMASTER. But there is a race: The record may have
+ * been migrated off the DMASTER while the redirected packet is
+ * on the wire (or in the local queue). So in case the record has
+ * migrated off the new destinaton of the call packet, instead of
+ * going back to the LMASTER to get the new DMASTER, we try to
+ * reduce rountrips by fist chasing the record a couple of times
+ * before giving up the direct chase and finally going back to the
+ * LMASTER (again). Note that this works because auf this: When
+ * a record is migrated off a node, then the new DMASTER is stored
+ * in the record's copy on the former DMASTER.
+ *
+ * The maxiumum number of attempts for direct chase to make before
+ * going back to the LMASTER is configurable by the tunable
+ * "MaxRedirectCount".
+ */
static void ctdb_call_send_redirect(struct ctdb_context *ctdb,
TDB_DATA key,
struct ctdb_req_call *c,
@@ -160,7 +181,7 @@ static void ctdb_send_dmaster_reply(struct ctdb_db_context *ctdb_db,
tmp_ctx = talloc_new(ctdb);
/* send the CTDB_REPLY_DMASTER */
- len = offsetof(struct ctdb_reply_dmaster, data) + key.dsize + data.dsize;
+ len = offsetof(struct ctdb_reply_dmaster, data) + key.dsize + data.dsize + sizeof(uint32_t);
r = ctdb_transport_allocate(ctdb, tmp_ctx, CTDB_REPLY_DMASTER, len,
struct ctdb_reply_dmaster);
CTDB_NO_MEMORY_FATAL(ctdb, r);
@@ -173,6 +194,7 @@ static void ctdb_send_dmaster_reply(struct ctdb_db_context *ctdb_db,
r->db_id = ctdb_db->db_id;
memcpy(&r->data[0], key.dptr, key.dsize);
memcpy(&r->data[key.dsize], data.dptr, data.dsize);
+ memcpy(&r->data[key.dsize+data.dsize], &header->flags, sizeof(uint32_t));
ctdb_queue_packet(ctdb, &r->hdr);
@@ -201,13 +223,18 @@ static void ctdb_call_send_dmaster(struct ctdb_db_context *ctdb_db,
return;
}
+ if (data->dsize != 0) {
+ header->flags |= CTDB_REC_FLAG_MIGRATED_WITH_DATA;
+ }
+
if (lmaster == ctdb->pnn) {
ctdb_send_dmaster_reply(ctdb_db, header, *key, *data,
c->hdr.srcnode, c->hdr.reqid);
return;
}
- len = offsetof(struct ctdb_req_dmaster, data) + key->dsize + data->dsize;
+ len = offsetof(struct ctdb_req_dmaster, data) + key->dsize + data->dsize
+ + sizeof(uint32_t);
r = ctdb_transport_allocate(ctdb, ctdb, CTDB_REQ_DMASTER, len,
struct ctdb_req_dmaster);
CTDB_NO_MEMORY_FATAL(ctdb, r);
@@ -220,6 +247,7 @@ static void ctdb_call_send_dmaster(struct ctdb_db_context *ctdb_db,
r->datalen = data->dsize;
memcpy(&r->data[0], key->dptr, key->dsize);
memcpy(&r->data[key->dsize], data->dptr, data->dsize);
+ memcpy(&r->data[key->dsize + data->dsize], &header->flags, sizeof(uint32_t));
header->dmaster = c->hdr.srcnode;
if (ctdb_ltdb_store(ctdb_db, *key, header, *data) != 0) {
@@ -237,10 +265,10 @@ static void ctdb_call_send_dmaster(struct ctdb_db_context *ctdb_db,
must be called with the chainlock held. This function releases the chainlock
*/
-static void ctdb_become_dmaster(struct ctdb_db_context *ctdb_db,
+static void ctdb_become_dmaster(struct ctdb_db_context *ctdb_db,
struct ctdb_req_header *hdr,
TDB_DATA key, TDB_DATA data,
- uint64_t rsn)
+ uint64_t rsn, uint32_t record_flags)
{
struct ctdb_call_state *state;
struct ctdb_context *ctdb = ctdb_db->ctdb;
@@ -252,6 +280,21 @@ static void ctdb_become_dmaster(struct ctdb_db_context *ctdb_db,
ZERO_STRUCT(header);
header.rsn = rsn + 1;
header.dmaster = ctdb->pnn;
+ header.flags = record_flags;
+
+ state = ctdb_reqid_find(ctdb, hdr->reqid, struct ctdb_call_state);
+
+ if (state) {
+ if (state->call->flags & CTDB_CALL_FLAG_VACUUM_MIGRATION) {
+ /*
+ * We temporarily add the VACUUM_MIGRATED flag to
+ * the record flags, so that ctdb_ltdb_store can
+ * decide whether the record should be stored or
+ * deleted.
+ */
+ header.flags |= CTDB_REC_FLAG_VACUUM_MIGRATED;
+ }
+ }
if (ctdb_ltdb_store(ctdb_db, key, &header, data) != 0) {
ctdb_fatal(ctdb, "ctdb_reply_dmaster store failed\n");
@@ -263,7 +306,6 @@ static void ctdb_become_dmaster(struct ctdb_db_context *ctdb_db,
return;
}
- state = ctdb_reqid_find(ctdb, hdr->reqid, struct ctdb_call_state);
if (state == NULL) {
DEBUG(DEBUG_ERR,("pnn %u Invalid reqid %u in ctdb_become_dmaster from node %u\n",
@@ -324,12 +366,19 @@ void ctdb_request_dmaster(struct ctdb_context *ctdb, struct ctdb_req_header *hdr
TDB_DATA key, data, data2;
struct ctdb_ltdb_header header;
struct ctdb_db_context *ctdb_db;
+ uint32_t record_flags = 0;
+ size_t len;
int ret;
key.dptr = c->data;
key.dsize = c->keylen;
data.dptr = c->data + c->keylen;
data.dsize = c->datalen;
+ len = offsetof(struct ctdb_req_dmaster, data) + key.dsize + data.dsize
+ + sizeof(uint32_t);
+ if (len <= c->hdr.length) {
+ record_flags = *(uint32_t *)&c->data[c->keylen + c->datalen];
+ }
ctdb_db = find_ctdb_db(ctdb, c->db_id);
if (!ctdb_db) {
@@ -386,10 +435,13 @@ void ctdb_request_dmaster(struct ctdb_context *ctdb, struct ctdb_req_header *hdr
/* use the rsn from the sending node */
header.rsn = c->rsn;
+ /* store the record flags from the sending node */
+ header.flags = record_flags;
+
/* check if the new dmaster is the lmaster, in which case we
skip the dmaster reply */
if (c->dmaster == ctdb->pnn) {
- ctdb_become_dmaster(ctdb_db, hdr, key, data, c->rsn);
+ ctdb_become_dmaster(ctdb_db, hdr, key, data, c->rsn, record_flags);
} else {
ctdb_send_dmaster_reply(ctdb_db, &header, key, data, c->dmaster, hdr->reqid);
@@ -562,6 +614,8 @@ void ctdb_reply_dmaster(struct ctdb_context *ctdb, struct ctdb_req_header *hdr)
struct ctdb_reply_dmaster *c = (struct ctdb_reply_dmaster *)hdr;
struct ctdb_db_context *ctdb_db;
TDB_DATA key, data;
+ uint32_t record_flags = 0;
+ size_t len;
int ret;
ctdb_db = find_ctdb_db(ctdb, c->db_id);
@@ -574,6 +628,11 @@ void ctdb_reply_dmaster(struct ctdb_context *ctdb, struct ctdb_req_header *hdr)
key.dsize = c->keylen;
data.dptr = &c->data[key.dsize];
data.dsize = c->datalen;
+ len = offsetof(struct ctdb_reply_dmaster, data) + key.dsize + data.dsize
+ + sizeof(uint32_t);
+ if (len <= c->hdr.length) {
+ record_flags = *(uint32_t *)&c->data[c->keylen + c->datalen];
+ }
ret = ctdb_ltdb_lock_requeue(ctdb_db, key, hdr,
ctdb_call_input_pkt, ctdb, False);
@@ -585,7 +644,7 @@ void ctdb_reply_dmaster(struct ctdb_context *ctdb, struct ctdb_req_header *hdr)
return;
}
- ctdb_become_dmaster(ctdb_db, hdr, key, data, c->rsn);
+ ctdb_become_dmaster(ctdb_db, hdr, key, data, c->rsn, record_flags);
}
diff --git a/server/ctdb_control.c b/server/ctdb_control.c
index 69724e3..748907f 100644
--- a/server/ctdb_control.c
+++ b/server/ctdb_control.c
@@ -604,6 +604,15 @@ static int32_t ctdb_control_dispatch(struct ctdb_context *ctdb,
CHECK_CONTROL_DATA_SIZE(0);
return ctdb_control_get_stat_history(ctdb, c, outdata);
+ case CTDB_CONTROL_SCHEDULE_FOR_DELETION: {
+ struct ctdb_control_schedule_for_deletion *d;
+ size_t size = offsetof(struct ctdb_control_schedule_for_deletion, key);
+ CHECK_CONTROL_MIN_DATA_SIZE(size);
+ d = (struct ctdb_control_schedule_for_deletion *)indata.dptr;
+ size += d->keylen;
+ CHECK_CONTROL_DATA_SIZE(size);
+ return ctdb_control_schedule_for_deletion(ctdb, indata);
+ }
default:
DEBUG(DEBUG_CRIT,(__location__ " Unknown CTDB control opcode %u\n", opcode));
return -1;
diff --git a/server/ctdb_daemon.c b/server/ctdb_daemon.c
index 9c650a0..75344ad 100644
--- a/server/ctdb_daemon.c
+++ b/server/ctdb_daemon.c
@@ -777,6 +777,7 @@ int ctdb_start_daemon(struct ctdb_context *ctdb, bool do_fork, bool use_syslog,
block_signal(SIGPIPE);
ctdbd_pid = getpid();
+ ctdb->ctdbd_pid = ctdbd_pid;
DEBUG(DEBUG_ERR, ("Starting CTDBD as pid : %u\n", ctdbd_pid));
diff --git a/server/ctdb_freeze.c b/server/ctdb_freeze.c
index 86cb5ed..0f70fd3 100644
--- a/server/ctdb_freeze.c
+++ b/server/ctdb_freeze.c
@@ -25,6 +25,7 @@
#include "../include/ctdb_private.h"
#include "lib/util/dlinklist.h"
#include "db_wrap.h"
+#include "../common/rb_tree.h"
static bool later_db(const char *name)
{
@@ -605,5 +606,15 @@ int32_t ctdb_control_wipe_database(struct ctdb_context *ctdb, TDB_DATA indata)
return -1;
}
+ if (!ctdb_db->persistent) {
+ talloc_free(ctdb_db->delete_queue);
+ ctdb_db->delete_queue = trbt_create(ctdb_db, 0);
+ if (ctdb_db->delete_queue == NULL) {
+ DEBUG(DEBUG_ERR, (__location__ " Failed to re-create "
+ "the vacuum tree.\n"));
+ return -1;
+ }
+ }
+
return 0;
}
diff --git a/server/ctdb_ltdb_server.c b/server/ctdb_ltdb_server.c
index 19a68ec..92fb0f6 100644
--- a/server/ctdb_ltdb_server.c
+++ b/server/ctdb_ltdb_server.c
@@ -25,6 +25,7 @@
#include "system/dir.h"
#include "system/time.h"
#include "../include/ctdb_private.h"
+#include "../common/rb_tree.h"
#include "db_wrap.h"
#include "lib/util/dlinklist.h"
#include <ctype.h>
@@ -49,6 +50,199 @@ static int ctdb_fetch_func(struct ctdb_call_info *call)
}
+/**
+ * write a record to a normal database
+ *
+ * This is the server-variant of the ctdb_ltdb_store function.
+ * It contains logic to determine whether a record should be
+ * stored or deleted. It also sends SCHEDULE_FOR_DELETION
+ * controls to the local ctdb daemon if apporpriate.
+ */
+static int ctdb_ltdb_store_server(struct ctdb_db_context *ctdb_db,
+ TDB_DATA key,
+ struct ctdb_ltdb_header *header,
+ TDB_DATA data)
+{
+ struct ctdb_context *ctdb = ctdb_db->ctdb;
+ TDB_DATA rec;
+ int ret;
+ bool seqnum_suppressed = false;
+ bool keep = false;
+ bool schedule_for_deletion = false;
+ uint32_t lmaster;
+
+ if (ctdb->flags & CTDB_FLAG_TORTURE) {
+ struct ctdb_ltdb_header *h2;
+ rec = tdb_fetch(ctdb_db->ltdb->tdb, key);
+ h2 = (struct ctdb_ltdb_header *)rec.dptr;
+ if (rec.dptr && rec.dsize >= sizeof(h2) && h2->rsn > header->rsn) {
+ DEBUG(DEBUG_CRIT,("RSN regression! %llu %llu\n",
+ (unsigned long long)h2->rsn, (unsigned long long)header->rsn));
+ }
+ if (rec.dptr) free(rec.dptr);
+ }
+
+ if (ctdb->vnn_map == NULL) {
+ /*
+ * Called from a client: always store the record
+ * Also don't call ctdb_lmaster since it uses the vnn_map!
+ */
+ keep = true;
+ goto store;
+ }
+
+ lmaster = ctdb_lmaster(ctdb_db->ctdb, &key);
+
+ /*
+ * If we migrate an empty record off to another node
+ * and the record has not been migrated with data,
+ * delete the record instead of storing the empty record.
+ */
+ if (data.dsize != 0) {
+ keep = true;
+ } else if (ctdb_db->persistent) {
+ keep = true;
+ } else if (header->flags & CTDB_REC_FLAG_AUTOMATIC) {
+ /*
+ * The record is not created by the client but
+ * automatically by the ctdb_ltdb_fetch logic that
+ * creates a record with an initial header in the
+ * ltdb before trying to migrate the record from
+ * the current lmaster. Keep it instead of trying
+ * to delete the non-existing record...
+ */
+ keep = true;
+ schedule_for_deletion = true;
+ } else if (header->flags & CTDB_REC_FLAG_MIGRATED_WITH_DATA) {
+ keep = true;
+ } else if (ctdb_db->ctdb->pnn == lmaster) {
+ /*
+ * If we are lmaster, then we usually keep the record.
+ * But if we retrieve the dmaster role by a VACUUM_MIGRATE
+ * and the record is empty and has never been migrated
+ * with data, then we should delete it instead of storing it.
+ * This is part of the vacuuming process.
--
CTDB repository
More information about the samba-cvs
mailing list