Notes toward an rsync regression suite

[Martin Pool]

It would help with rsync quality if we had an automatic test suite
that could run on the build farm machines to validate functionality.
Writing a test suite is hard when there is no requirements document.

The best thing to do is probably to start with the functionality that
is both commonly used and easy to test, and work up from there.  So in
the case of rsync this is copying from one directory to another, and
then the straightforward options like --exclude and --delete.

On the other hand you should test where you expect to find bugs.  So,
at the moment, this seems to be in the networking code rather than
--exclude.

Here are a few ideas:

 * The test must run unattended by 'make check'.  It should display
   some kind of progress indication.

 * It should be possible to run the tests without having root on the
   box, and without requiring any changes to configuration.

 * Running the tests should not interfere with other services on the
   system (such as an rsyncd) and should not open any security holes.
   So, if we listen on a TCP port we should choose a high-numbered one
   that is dynamically checked not to be in use.

 * Testing should not impose an undue load on the test machine.  So,
   on today's machines, using 10GB of disk to run the test suite will
   discourage many people from running it.  Similarly for VM and CPU
   usage.  Tests should probably run maximally nice.  Similarly the
   suite probably ought to run in less than 15minutes on typical
   hardware.  On the other hand there may be some bugs that only arise
   on very large transfers, so it should be possible to run them if
   desired.

 * If the test fails, the error message should show all the
   information that the developers are likely to want to know.  Tests
   should be as reproducible as possible, including reproduction
   across platforms.  So for example rather than random numbers, we
   should use pseudorandom numbers from a predictable sequence.

 * Testing across a network would be nice, but it is harder to
   coordinate.  Testing across a loopback TCP socket to localhost is
   adequate.

 * If facilities like running as root or having sshd are available on
   a particular box it would be good to take advantage of them.  We
   need to distinguish "not possible to test this" from "failed".

 * The test must terminate itself and fail if it does not complete in
   a reasonable amount of time.	  

 * It's acceptable to use a scripting language like Perl, Python or
   Tcl.  Tcl is probably less desirable than the other two because the
   language is a little ugly and not so many people know it these
   days.  Python is better.  Since rsync is more-or-less
   noninteractive using expect is not a big benefit as far as I can
   see.

 * Using only POSIX sh would be better because it will work on all
   build farm contributed machines.  I don't know if you can get
   Python or Perl on Cray UNICOS, for example.  You can imagine a
   hybrid system like autoconf where a high-level Python program
   generates a more longwinded sh script.  (And we all know how
   friendly autoconf is :-)

 * You can use DejaGNU or some other framework if you want.  Or not.
   The framework generated by automake is interesting, though we
   probably don't want to adopt automake.

 * Automated testing Nirvana is to be able to add a test that
   precisely fails on any reported bug, and that then passes when the
   bug is fixed.

I don't know if I'll get around to this soon, but if anybody feels
like trying they're very welcome.

-- 
Martin