superlifter design notes (was Re: Latest rZync release: 0.06)

[Martin Pool]

I'm inclined to agree with jw that truthfully representing time and
leap seconds is a problem for the operating system, not for us.  We
just need to be able to accurately represent whatever it tells us,
without thinking very much about the meaning.

Somebody previously pointed out that timestamp precision is not a
property of the kernel, but rather of the filesystem on which the
files are stored.  In general there may be no easy way to determine it
ahead of time: you can (if you squint) imagine a network filesystem
with nanosecond resolution that's served by something with rather
less.  I suspect the only way to know may be to set the time and then
read it back.

You can also imagine that in the next few years some platform may
change to a format that accurately represents leap seconds, whether by
TAI or something else.  (I'm not sure if I'd put money on it.)
Presumably that machine's POSIX interface will do a lossy conversion
back to regular Unix time to support old apps.  If we merely used that
information, then when replicating between two such machines, files
whose mtime happened to near on a leap second would be inaccurate.
That would contradict our goal of preserving precision as much as
possible, even if we can't tell if it is accurate.

Ideally, we would use the native interface so as to be able to get the
machine's full precision, and that would imply something like TAI
internally.

Whether this is worth doing depends on whether you reckon any platform
will actually move to a filesystem that can represent leap seconds.
As jw says, practically all machines have clocks with more than one
second of inaccuracy, so handling leap seconds is not practically
important.  Certainly they might use it within their ntp code, but I
don't know if they'll expose it to applications.

> What is the actual format of TAI?

64-bit signed seconds-since-1970, plus optionally nanoseconds, plus
optionally attoseconds.  (There's something rather fascinating about
using attoseconds.)

To be fair, it seems that TAI is an international standard, and djb
just made up libtai, not the whole thing.  (Mind you, from some
standards I've seen, that would be a good reason to walk briskly
away.)

One drawback, which is not realy djb's fault, is that if you
inadvertently use a TAI value as a Unix value it will be about 10
seconds off -- almost, but not quite, correct.  I'd hate to have bugs
like that but presumably they can be avoided by using the interface
correctly.

On the other hand, sint32 unix time is clearly running out, and if we
have to use something perhaps it might as well be TAI. 

I would kind of prefer just a single 64-bit quantity measured in (say)
nanoseconds, and compromise on being able to time the end of the
universe, but I don't think I care enough to invent a new standard.

-- 
Martin