What does this mean? select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)

Wed Jan 24 11:46:22 UTC 2018

I take it that:
- the file descriptors 1,2 here are either stdin or stdout 
(https://en.wikipedia.org/wiki/File_descriptor)
- rsync client - server communication goes via stdin, stdout
- servers are running ntpd

Concluding from the new logs below. 
There is a problem on the client starting around 05:08:22 (cannot read 
from the server?)
Then the backup server looses connection at 05:22:50 and writes this to 
stderr log 
And at 07:19:43 the client notices (times out) the server is gone and 
also logs the broken pipe.

Is it save to assume that the connection is dropped of this session?

At 05:22:50 this message is archived at the backup server:
packet_write_wait: Connection to 192.168.10.43 port 22: Broken pipe
rsync: connection unexpectedly closed (305252479 bytes received so far) 
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) 
[receiver=3.0.9]
rsync: connection unexpectedly closed (27810 bytes received so far) 
[generator]
rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9]

On the client debug I have this:
17383 05:08:22 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
{59, 999997})
17383 05:08:22 write(1, "\374\17\0\7", 4) = 4
17383 05:08:22 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
{59, 999997})
17383 05:08:22 write(1, "f=\nt=22 aria-labelledby=3D=22left=22 
bgcolor=3D=22=2386526f=22 border=3D=220=\n=22 cellpadding=3D=220"..., 
4092) = 4092
17383 05:08:22 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
{59, 999997})
17383 05:08:22 write(1, "\374\17\0\7", 4) = 4
17383 05:08:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:09:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:10:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:11:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:12:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:13:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:14:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:15:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 05:16:22 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)

17383 07:12:27 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:13:27 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:14:28 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:15:28 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:16:28 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:17:28 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:18:28 select(2, NULL, [1], [1], {60, 0}) = 0 (Timeout)
17383 07:19:28 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
{45, 169717})
17383 07:19:43 write(1, "f=\nt=22 aria-labelledby=3D=22left=22 
bgcolor=3D=22=2386526f=22 border=3D=220=\n=22 cellpadding=3D=220"..., 
4092) = -1 EPIPE (Broken pipe)
17383 07:19:43 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, 
si_pid=17383, si_uid=0} ---
17383 07:19:43 write(2, "rsync: writefd_unbuffered failed to write 4092 
bytes to socket [sender]: Broken pipe (32)", 89) = 89
17383 07:19:43 write(2, "\n", 1)        = 1
17383 07:19:43 rt_sigaction(SIGUSR1, {SIG_IGN, [], SA_RESTORER, 
0x7fc3ad2c6270}, NULL, 8) = 0
17383 07:19:43 rt_sigaction(SIGUSR2, {SIG_IGN, [], SA_RESTORER, 
0x7fc3ad2c6270}, NULL, 8) = 0
17383 07:19:43 write(2, "rsync error: error in rsync protocol data 
stream (code 12) at io.c(1532) [sender=3.0.9]", 87) = 87
17383 07:19:43 write(2, "\n", 1)        = 1
17383 07:19:43 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
17383 07:19:43 exit_group(12)           = ?
17383 07:19:43 +++ exited with 12 +++

-----Original Message-----
From: Kevin Korb via rsync [mailto:rsync at lists.samba.org] 
Sent: maandag 22 januari 2018 17:10
To: rsync at lists.samba.org
Subject: Re: What does this mean? select(1, [0], [], NULL, {60, 0}) = 0 
(Timeout)

From man 2 select:
int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set 
*exceptfds, struct timeval *timeout);

So, it is waiting for file descriptor #1 to become available with a 60 
second timeout which it is hitting.

Use lsof to find out what file descriptor #1 is.

On 01/22/2018 10:07 AM, Marc Roos via rsync wrote:
> 
> I have been having broken pipe messages for a while. I think since a 
> network architecture change. But I am not sure because I have not been 

> monitoring the errors closely before. I added this
> (https://rsync.samba.org/issues.html) rsync-debug.sh script with 
> strace of 100. But I have no idea how to interper this output.
> 
> I guess between 00:55:41 and 03:06:47 nothing is happening? And then 
> some timeout is generated? What does this 'select(1, [0], [], NULL, 
> {60,
> 0}) = 0 (Timeout)' mean?
>  
> 
> 
> 
> 
> Commandline
> ===========
> rsync -avH --del --rsync-path=/usr/local/sbin/rsync-debug.sh
> --one-file-system --exclude-from=/home/backup/excluded-dirs
> --include-from=/home/backup/included-dirs --timeout=0 --ignore-errors 
> --numeric-ids -e "ssh -o Compression=no -x -o StrictHostKeyChecking=no 

> -b 192.168.10.57" --link-dest=../../../x/x/x/ SRC DST >>  backup-2.log
> 2>> error.log
> 
> 
> RECEIVING DATA SERVER
> =====================
> packet_write_wait: Connection to 192.168.30.43 port 22: Broken pipe
> rsync: connection unexpectedly closed (72833675 bytes received so far) 

> [receiver]
> rsync error: error in rsync protocol data stream (code 12) at 
io.c(605) 
> [receiver=3.0.9]
> rsync: connection unexpectedly closed (15058 bytes received so far) 
> [generator]
> rsync error: unexplained error (code 255) at io.c(605) 
[generator=3.0.9]
> 
> SENDING DATA SERVER
> ===================
> 10137 00:55:40 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{59, 
> 999997})
> 10137 00:55:40 read(0, 
> 
"m\306Y{za\332D[Ko\254R\25\226\2H)a\34e\342\rrE\376\212\240-\214;\224@\3
> 
75+\266s\f\217\344\365\233\205`\352Vb\271r=.\260:\3179\376\367\352\337\3
> 
72T\237^\242\227\0\273kCr/\322#\360|\214\316\222(\314`3X\2051Id\223z/\26
> N\211\360\30\236\205%Y\355"..., 4092) = 4092
> 10137 00:55:40 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{59, 
> 943902})
> 10137 00:55:41 read(0, "\374\17\0\7", 4) = 4
> 10137 00:55:41 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{59, 
> 999997})
> 10137 00:55:41 read(0, 
> 
"\25\\\223R\233\263\355wW\23\203;4\212\214\331>\212\251\333=e\237\"\307r
> 
\313\34\273M\275Y\10\242\367\273\7\2679P\332F\312q\240eL)\356\204\17\203
> 
k\375}\306\331\256\357\20B\326kP\236\240\10\222c\303\261k\7\220\2040\26l
> 6X\343\2\234i\275\244\243\300\222\302\177\245(fN\270\323\376\331Y"..., 

> 4092) = 4092
> 10137 00:55:41 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{59, 
> 990075})
> 10137 00:55:41 read(0, "\374\17\0\7", 4) = 4
> 10137 00:55:41 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{59, 
> 999997})
> 10137 00:55:41 read(0, 
> 
"\361Zu\267\370\10\227\331\3431\3521\222\340g\370o\330\371\f\327\331\257
> 
\321\22\33y\301Uq\307\302s\"\n\213\332}]\272\217\244\221\301|\\\240V]\32
> 
4L\4\362\231'\365X\276^\5\375\235\25\364\320\337\274\267;\27*\352\227\21
> 0h\315\243\372\267\310c-n\225\374V\315o 
\350\6\350\365\234\37f:q\5s"..., 
> 4092) = 4092
> 10137 00:55:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 00:56:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 00:57:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 00:58:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 00:59:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:00:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:01:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:02:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:03:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:04:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:05:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:06:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:07:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:08:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:09:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:10:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:11:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:12:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:13:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:14:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:15:41 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:16:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:17:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:18:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:19:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:20:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:21:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:22:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:23:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:24:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:25:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:26:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:27:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:28:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:29:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:30:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:31:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:32:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:33:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:34:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:35:42 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:36:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:37:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:38:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:39:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:40:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:41:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:42:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:43:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:44:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:45:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:46:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:47:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:48:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:49:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:50:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:51:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:52:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:53:43 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:54:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:55:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:56:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:57:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:58:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 01:59:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:00:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:01:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:02:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:03:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:04:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:05:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:06:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:07:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:08:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:09:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:10:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:11:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:12:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:13:44 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:14:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:15:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:16:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:17:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:18:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:19:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:20:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:21:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:22:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:23:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:24:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:25:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:26:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:27:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:28:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:29:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:30:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:31:45 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:32:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:33:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:34:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:35:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:36:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:37:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:38:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:39:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:40:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:41:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:42:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:43:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:44:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:45:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:46:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:47:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:48:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:49:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:50:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:51:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:52:46 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:53:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:54:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:55:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:56:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:57:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:58:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 02:59:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:00:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:01:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:02:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:03:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:04:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:05:47 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
> 10137 03:06:47 select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left 
{47, 
> 963418})
> 10137 03:06:59 read(0, "", 4)           = 0
> 10137 03:06:59 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
> {59, 999997})
> 10137 03:06:59 write(1, "O\0\0\nrsync: connection unexpectedly closed 
> (4828392 bytes received so far) [sender]\n", 83) = 83
> 10137 03:06:59 rt_sigaction(SIGUSR1, {SIG_IGN, [], SA_RESTORER, 
> 0x7f4111266270}, NULL, 8) = 0
> 10137 03:06:59 rt_sigaction(SIGUSR2, {SIG_IGN, [], SA_RESTORER, 
> 0x7f4111266270}, NULL, 8) = 0
> 10137 03:06:59 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left 
> {59, 999997})
> 10137 03:06:59 write(1, "W\0\0\nrsync error: error in rsync protocol 
> data stream (code 12) at io.c(605) [sender=3.0.9]\n", 91) = 91
> 10137 03:06:59 select(0, NULL, NULL, NULL, {0, 100000}) = 0 (Timeout)
> 10137 03:06:59 exit_group(12)           = ?
> 10137 03:06:59 +++ exited with 12 +++
> 
> 
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. 
> F1 Outsourcing Development Sp. z o.o.
> Poland 
> 
> t:  +48 (0)124466845
> f:  +48 (0)124466843
> e:  marc at f1-outsourcing.eu
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,