TDB ever-decreasing store rate

Andy Smith andy at
Mon May 12 02:56:07 GMT 2003

Hi guys,

Is this the correct list to ask about TDB?  If not, some hint as to
where might be more appropriate would be appreciated.

I've got around 600,000 rows of data in a MySQL table which I need
to periodically put into a TDB file.  Only two columns are needed, an
IP address and a time stamp.  I have written a small program in C to
get the data out of MySQL and store each row in a TDB, storing the
IP as a 32bit int and the timestamp also as a 32bit int (time_t).

The problem is, the rate of insertions I am getting becomes quite
pitiful as the amount of data I try to store increases.  I _am_
doing this on a very low-spec machine, a mere AMD K6 500, but I
would expect a little more than the 10 or so tdb_store()s that I
appear to be able to do per second (so yes, it's now taking more
than 3 hours to do this).  The results look like this:

is that normal?

This is absolutely not anything to do with the time to grab the data
out of MySQL.  I have verified this by commenting out the
tdb_store() line, in which case the code completes within a very
short time directly proportional to the number of rows retrieved.

Is there any way I can make this faster?  Should I be using a
specific hash value on the tdb_open(), if I know how many entries
there will be in the final TDB?

Is there any way I could keep the TDB is memory until it is complete
and then write it out to disk?

Or perhaps TDB is not what I want to be using here.. all I need is a
database I can write to a file, rsync it to a bunch of hosts and
have multiple readers - once the file is written out, no further
writes will be done.

Here's the tdb_open() line I'm currently using:

tdb = tdb_open(TDB_FILE1, 0, 0, O_RDWR|O_CREAT, 0600);

and here's the tdb_store() one:

if (tdb_store(tdb, key, data, TDB_REPLACE) == -1) {
        /* error stuffs here */

If the full code of my little test program would help, I'll put it
up somewhere.

Any help would be really appreciated.


More information about the samba-technical mailing list