TDB ever-decreasing store rate

[Andy Smith]

Hi guys,

Is this the correct list to ask about TDB?  If not, some hint as to
where might be more appropriate would be appreciated.

I've got around 600,000 rows of data in a MySQL table which I need
to periodically put into a TDB file.  Only two columns are needed, an
IP address and a time stamp.  I have written a small program in C to
get the data out of MySQL and store each row in a TDB, storing the
IP as a 32bit int and the timestamp also as a 32bit int (time_t).

The problem is, the rate of insertions I am getting becomes quite
pitiful as the amount of data I try to store increases.  I _am_
doing this on a very low-spec machine, a mere AMD K6 500, but I
would expect a little more than the 10 or so tdb_store()s that I
appear to be able to do per second (so yes, it's now taking more
than 3 hours to do this).  The results look like this:

http://strugglers.net/tdb.png

is that normal?

This is absolutely not anything to do with the time to grab the data
out of MySQL.  I have verified this by commenting out the
tdb_store() line, in which case the code completes within a very
short time directly proportional to the number of rows retrieved.

Is there any way I can make this faster?  Should I be using a
specific hash value on the tdb_open(), if I know how many entries
there will be in the final TDB?

Is there any way I could keep the TDB is memory until it is complete
and then write it out to disk?

Or perhaps TDB is not what I want to be using here.. all I need is a
database I can write to a file, rsync it to a bunch of hosts and
have multiple readers - once the file is written out, no further
writes will be done.

Here's the tdb_open() line I'm currently using:

tdb = tdb_open(TDB_FILE1, 0, 0, O_RDWR|O_CREAT, 0600);

and here's the tdb_store() one:

if (tdb_store(tdb, key, data, TDB_REPLACE) == -1) {
        /* error stuffs here */
}

If the full code of my little test program would help, I'll put it
up somewhere.

Any help would be really appreciated.

Thanks!
Andy