rsync-2.4.7 NEWS file

[Martin Pool]

On  6 Aug 2001, "John E. Malmberg" <malmberg at Encompasserve.org> wrote:

> I look forward to seeing the "pre" tarballs.

The first one is up now, in the rsync.samba.org ftp area.

> I am starting to port RSYNC to OpenVMS, and with 2.4.6 there are a
> number of issues that it would help me if they were fixed.

OK, I think it would be good to have support there too.

It would be ideal if we could get an OpenVMS machine into
build.samba.org, so that people can avoid breaking rsync on it in the
future.  This has helped a great deal with platforms like Cray UNICOS
to which most of us don't have regular access and which are
considerably unlike PCs.  Of course this requires having rsync working
to start with... 

> Because of being a bit internet impaired, I can not supply diff files
> until tomorrow, and I am going from my memory, and it may be better if
> I apply my changes to 2.4.7pre instead.

If you can't do diffs, then it's OK to just post a tarball or zip and
I'll make up the diff myself.

> I currently have the source code (with some patches) compiling cleanly
> with the Compaq C compiler in a -WALL mode.

Well done!

> The first issue that comes up is that at this time, I can not support
> hardlinks.

OK.  One of the cleanups in 2.4.7 was to try to work better on
machines like this.  It may not be complete.  I look forward to seeing
what was necessary to make it work on VMS.

> Some of the code will not honor the config.h's settings on this, and will
> compile in references to other routines that are not compiled as they are
> honoring config.h.
> 
> This would lead me to beleive it is a bug.

That sounds like a bug.

> I also do not have the UNIX SYSLOG API.
> The SYSLOG checks also continue to expect some of the SYSLOG definitions
> to be present even when CONFIG.H says otherwise.

OK.   We should fix that too. 

"Untested code is broken code."

> Compaq C will not compile code where a function prototype of a
> standard library call is placed in the code that is not type compatable
> with the standard headers that were previously #included.
> 
> This compiler is common to OpenVMS, Tru-64, and LINUX on Alpha.

Right.  We shouldn't be doing that.

> The Compaq C compiler on OpenVMS by default has char as a signed type.
> The rsync code seems to be generally assuming that char is unsigned.
> The compiler will flag a type mismatch when attempting to pass an
> (unsigned char) to a (char) procedure parameter or assignment.  On
> examining the code where these were flagged, it appears that a number of
> char arrays where really being used as unsinged char arrays.

OK.  We shouldn't depend on that.

> Several of the prototypes of functions with no parameters were missing the
> (void) tag.

That should be fixed too.  I don't think rsync needs to support
pre-ANSI compilers.

> A few cases, most notably the return from a strlen() call, the code is
> testing to see if it is greater than 0.  As the size_t type is unsigned,
> it is not possible for this value to be less than 0, so the compiler is
> concerned that (unsinged > 0) might fail to detect a routine that returns
> a negative value cast to an unsigned value.  A change to (unsigned != 0)
> clarifies this.  The compiler generates the same output code in either
> case, but the second expression makes it clear that the only real test is
> for non-zero.

Right, that would be good to fix.  gcc does that test too, but perhaps
we don't have enough warning glags turned on.

> The Compaq C compiler is also flaging all #if statements that reference an
> undefined macro symbol in their line.

Right.  We should be clear about using #ifdef or #if as appropriate.

> There are two OpenVMS issues for compiling.
> 
> One is that my build environment is using a search list, to keep my VMS
> specific patched files separate from a read only directory tree containing
> the UNIX rsync source.

OK.
 
> Once consequence of this is that the compiler can not handle include files
> of the type "..\foo.h".  I can deal with "foo.h" and tell the compiler
> where find the files, or I can use "zlib\foo.h" and tell the compiler
> where zlib is.

I'm afraid I don't understand the problem.  Are you saying that we
should not have lines like <zlib/foo.h>, or just not <../foo.h>?  The
second is pretty crufty and I'd be happy to see it gone.

> The other is more tricky, as those of you that have read my posts on the
> SAMBA technical list have seen, OpenVMS has a 48 bit inode.  Since at the
> time C was introduced to OpenVMS, only 32 bit integers were available,
> this inode was represented in the struct stat as unsigned short
> st_ino[3].

!!!

> As a result all assignments and comparisons for inodes must be done
> through memcpy() and memcmp().  I have worked out macros that should
> resolve to memcpy() and memcmp() on OpenVMS, but resolve to normal code on
> UNIX platforms.  I do not know of any other platform with an inode type
> that can be larger than the largest integer value on the platform.
> 
> All of these patches that I have made to 2.4.6 were done from editor
> scripts, so I should be able to unpack 2.4.7pre and quickly see how that
> build gos.

OK, I don't see any harm in using those macros.

> I still do not have a runable executable, as I have to deal with
> converting the forked process to an OpenVMS AST thread, and a few other
> issues with file formats.  So this means that for execution purposes,
> my patches are untested.

If you could point me at a short description of what this means
relative to the Unix process model that would be very helpful for
rsync 3.0.  At this point I want to get away from the 3-process model
and probably use a select() loop to drive the network, with plain
blocking IO for disk.  I'm prepared to lose a bit of performance on
slow links or disks for the sake of scalability and a saner
architecture.

> >     * Change to using libpopt rather than getopt for processing
> >       options.  This makes the code cleaner and the behaviour more
> >       consistent across platforms.  popt is included and built if not
> >       installed on the platform.
> 
> Note, I have a modified gnu getopt() that handles commands in a more
> OpenVMS friendly format.  I will have to see if this is a step backwards
> for me.

It may be hard to convert.  It's not impossible to go back to getopt I
suppose, but at this point VMS looks like a niche audience.

> The basic differences are that OpenVMS users are used to using "/" where
> the gnu code uses "--", and that "foo=value" can be also specified as
> "foo:value".

popt is pretty configurable.  At the risk of making your life more
difficult, perhaps a patch to popt would be appropriate if it works in
other respects?

> I also have some Compaq OpenVMS specific issues to resolve:
> 
> 1. I see references to uid and gid's of -1, and -2.  Are these flags
>    the program, or do they map to a specific special UNIX user or
>    group?

They are Unix conventions for "nobody".  I think it's ugly to hardcode
the values, but that's the convention.  I think in 2.4.7 they're a
little clearer.  If you have a good way to change to "nobody" on VMS,
or if anybody would like to suggest a better way on Unix I'm happy to
hear it.

These are only used (from memory) if you don't specify another userid
for the daemon.

> 2. OpenVMS has a variety of internal file formats.  It can send out the
>    formats commonly used for text and binary files in a format that
>    UNIX can use them.  When receiving an update to an existing file I
>    think I can find a way to make sure that the orignal internal file
>    format is used.
> 
>    However, this conversion process is lossy.  The format information is
>    lost.  Also some of the OpenVMS file types are unsuable if this format
>    information is lost.  The types that are unusable would not normally
>    be readable on a non-OpenVMS host.
> 
>    It is for when two OpenVMS hosts are using RSYNC, I need to be able to
>    identify this, so that I can send that information to the receiver.
>    This would allow rsync to be fully functional between two OpenVMS
>    systems.
> 
>    For files that are obviously not text files, and therefore not portable
>    between operating systems, I was going to append the file format
>    information to the end of the file name, with some character (not yet
>    chosen) to indicate a delimiter.  When copied to a non-OpenVMS system,
>    they would have a funny name, but if that RSYNC server then served them
>    to another OpenVMS machine, it would then be able to correctly open
>    the file.

That sounds reasonable to me.  I suppose the main question is whether
that would be surprising behaviour to people used to VMS.

>    If someone has suggestions for a character to delimit the file from
>    it's OpenVMS attributes, I am interested.  I am leaning toward the
>    exclamation mark as it is a comment flag to the OpenVMS shell.
> 
>    If there are other suggestions on how a file server can give what
>    hints it knows about the use of a file it is serving on a portable
>    nature, I am also interested.

NFSv4, which is likely to be the underlying rpc transport for rsync,
uses a MIME Content-Type field for files.  Would that be sufficient?
I suppose not.

>    There may be other issues I come up with as well.  The OpenVMS file
>    system does the wildcard expansion, not the shell.  This changes
>    the efficiencies of some algorthms.

Happy hacking!

-- 
Martin 
VA Linux Systems