rsync-2.4.7 NEWS file

[John E. Malmberg]

It was discovering the SAMBA BUILD farm that got me off on the rsync
tangent.

On Tue, 7 Aug 2001, Martin Pool wrote:

> On  6 Aug 2001, "John E. Malmberg" <malmberg at Encompasserve.org> wrote:
> 
> > I look forward to seeing the "pre" tarballs.
> 
> The first one is up now, in the rsync.samba.org ftp area.

I will grab that A.S.A.P.
 
> > Once consequence of this is that the compiler can not handle include files
> > of the type "..\foo.h".  I can deal with "foo.h" and tell the compiler
> > where find the files, or I can use "zlib\foo.h" and tell the compiler
> > where zlib is.
> 
> I'm afraid I don't understand the problem.  Are you saying that we
> should not have lines like <zlib/foo.h>, or just not <../foo.h>?  The
> second is pretty crufty and I'd be happy to see it gone.

My preference is for "foo.h".  I can deal with most cases of "zlib/foo.h"
by establishing a temporary logical name of "zlib" that resolves to all
locations that "foo.h" might be found, listed in order of precedence.

Think of it as a symlink established by a shell that is only valid inside
the shell and the programs it launches, and self destructs automatically
after it is no longer needed.

But because I am following a path alias to the foobar.c file, the ".."
is not meaningful to find the parent directory.  There simply is not one
for it to find.

> > I still do not have a runable executable, as I have to deal with
> > converting the forked process to an OpenVMS AST thread, and a few other
> > issues with file formats.  So this means that for execution purposes,
> > my patches are untested.
> 
> If you could point me at a short description of what this means
> relative to the Unix process model that would be very helpful for
> rsync 3.0.  At this point I want to get away from the 3-process model
> and probably use a select() loop to drive the network, with plain
> blocking IO for disk.  I'm prepared to lose a bit of performance on
> slow links or disks for the sake of scalability and a saner
> architecture.

An AST is a software interrupt on an event.  OpenVMS's native API is all
asynchronous.

I guess that the closest I have seen in UNIX is the signal trap API.

For simplified example, if I need to create a file in the background, and
signal the main line when it is done:

I call the sys$open() call, and pass it the address of the routine to
execute when the sys$open() completes.  This routine would set up a
SYS$PUT() to put data in the file, and the SYS$PUT() would also receive
the address of the routine to do the next step.  Finally the last step
would set a global value to the main line to indicate it was done.

Note that the sys$open() call returns to it's caller immediately and
all the other processing happens in the background.

Because it is all inside one process, there is no performance penaltly of
the fork() call.  The closest OpenVMS equivalent of fork() is extremely
high overhead compared to a UNIX fork(), but use of the AST routines give
the overhead advantage to OpenVMS.

When an AST routine is running, it blocks all other ASTs from running, and
it blocks the mainline.  So some care is needed for the programming.

If you can map this to a UNIX concept, it would be great.  Especially if
it is something that can be easily  #ifdef around.

For use of the AST concept of SAMBA, it would take a lot more work than
RSYNC, as there is one security profile for the entire process.  It may be
possible to do this for nmbd.

The other issue with fork() to exec() a routine on OpenVMS is that the
overhead to create the second process is high enough, that most native
programs keep the second process around and just feed it additional
commands instead of creating a new process each time.

> popt is pretty configurable.  At the risk of making your life more
> difficult, perhaps a patch to popt would be appropriate if it works in
> other respects?

I will look into it.  If popt is becoming a standard feature on UNIX,
I can look at providing an OpenVMS specific taylored version.
 
> > I also have some Compaq OpenVMS specific issues to resolve:
> > 
> > 1. I see references to uid and gid's of -1, and -2.  Are these flags
> >    the program, or do they map to a specific special UNIX user or
> >    group?
> 
> They are Unix conventions for "nobody".  I think it's ugly to hardcode
> the values, but that's the convention.  I think in 2.4.7 they're a
> little clearer.  If you have a good way to change to "nobody" on VMS,
> or if anybody would like to suggest a better way on Unix I'm happy to
> hear it.

I guess I have to look at how they are used in the code.  A file on
OpenVMS can be owned by a non-existant user or resource.  Bad practice
but legal.  I expect that the same can be true with other platforms.
 
> These are only used (from memory) if you don't specify another userid
> for the daemon.
 
> > 2. OpenVMS has a variety of internal file formats.  It can send out the
> >    formats commonly used for text and binary files in a format that
> >    UNIX can use them.  When receiving an update to an existing file I
> >    think I can find a way to make sure that the orignal internal file
> >    format is used.
> > 
> >    However, this conversion process is lossy.  The format information is
> >    lost.  Also some of the OpenVMS file types are unsuable if this format
> >    information is lost.  The types that are unusable would not normally
> >    be readable on a non-OpenVMS host.
> > 
> >    It is for when two OpenVMS hosts are using RSYNC, I need to be able to
> >    identify this, so that I can send that information to the receiver.
> >    This would allow rsync to be fully functional between two OpenVMS
> >    systems.
> > 
> >    For files that are obviously not text files, and therefore not portable
> >    between operating systems, I was going to append the file format
> >    information to the end of the file name, with some character (not yet
> >    chosen) to indicate a delimiter.  When copied to a non-OpenVMS system,
> >    they would have a funny name, but if that RSYNC server then served them
> >    to another OpenVMS machine, it would then be able to correctly open
> >    the file.
> 
> That sounds reasonable to me.  I suppose the main question is whether
> that would be surprising behaviour to people used to VMS.

They would only see that filename if they were on a non-OpenVMS system.
It is the UNIX user that would get a surprise if I set up a rsync server
with OpenVMS backup savesets, intended for a remote OpenVMS site.

For keeping source trees in rsync, I would recommend that the OpenVMS
RSYNC server make sure that only portable file formats are supplied.
UNIX files are considered STREAM-LF on the OpenVMS side.

> >    If there are other suggestions on how a file server can give what
> >    hints it knows about the use of a file it is serving on a portable
> >    nature, I am also interested.
> 
> NFSv4, which is likely to be the underlying rpc transport for rsync,
> uses a MIME Content-Type field for files.  Would that be sufficient?
> I suppose not.

I will not know untill I start transferring files. :-)
 
> >    There may be other issues I come up with as well.  The OpenVMS file
> >    system does the wildcard expansion, not the shell.  This changes
> >    the efficiencies of some algorthms.
> 
> Happy hacking!
Thanks

-John
wb8tyw at qsl.network
Personal Opinion Only