HP-UX 11i and largefiles on rsync 2.6.2

Don Malloy nixuser23 at hotmail.com
Mon Jul 19 17:13:55 GMT 2004


Every which way I try, it gets to 2146336768 bytes and stops. I have to 
cancel it with CRTL-C - it doesn't matter if it's over a remsh, ssh or local 
filesystem.  There is nothing in the syslog regarding this. I have attached 
logs for all three (remsh,ssh, local fs - note local fs is much faster after 
the patch to generator.c)

I tried running an strace (mentioned in another post), however strace -f 
.... doesn't work in HPUX - the syntax is different.

Do you have any suggestions as to how to run the strace?, here is part of 
the man page:

strace(1M)

NAME
      strace - write STREAMS event trace messages to standard output

SYNOPSIS
      strace [ mod sub pri ] ...

DESCRIPTION
      strace gets STREAMS event trace messages from STREAMS drivers and
      modules via the STREAMS log driver (strlog(7)), and writes these
      messages to standard output.  By default, strace without arguments
      writes all STREAMS trace messages from all drivers and modules.
      strace with command-line arguments limits the trace messages received.

      The arguments, which must be specified in groups of three, are:

           mod  Specifies the STREAMS module identification number from the
                streamtab entry.

           sub  Specifies a subidentification number (often corresponding to
                a minor device).

           pri  Specifies a tracing priority level.  strace gets messages of
                a level equal to or less than the value specified by pri.
                Only positive integer values are allowed.

      The value all can be used for any argument in the strace command line
      to indicate that there are no restrictions for that argument.

      Multiple sets of the three arguments can be specified to obtain the
      messages from more than one driver or module.

      Only one strace process can open the STREAMS log driver at a time.

      When strace is invoked, the log driver compares the sets of command
      line arguments with actual trace messages, returning only messages
      that satisfy the specified criteria.

      STREAMS event trace messages have the following format:

           seq time tick pri ind mod sub text

----------------------------

Here is the output of the rsync via remsh:

server2 # /tmp/rsync -avvvvv -e remsh --rsync-path=/tmp/rsync --progress 
--stats file1.db server1:/spare
cmd=remsh machine=server1 user= path=/spare
cmd=remsh server1 /tmp/rsync --server -vvvvvlogDtpr . /spare
opening connection using remsh server1 /tmp/rsync --server -vvvvvlogDtpr . 
/spare
(Client) Protocol versions: remote=28, negotiated=28
(Server) Protocol versions: remote=28, negotiated=28
building file list ...
[sender] make_file(file1.db,*,2)
[sender] expand file_list to 131072 bytes, did move
1 file to consider
[sender] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 gid=3
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=24806
recv_file_name(file1.db)
received 1 names
gid 3(sys) maps to 3
[receiver] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 
gid=3
recv_file_list done
get_local_name count=1 /spare
recv_files(1) starting
generator starting pid=24806 count=1
delta transmission enabled
recv_generator(file1.db,0)
send_files(0, file1.db)
count=0 n=0 rem=0
send_files mapped file1.db of size 4304863232
calling match_sums file1.db
file1.db
2146336768  49%    3.04MB/s    0:11:32 (Press CTRL-C here)
_exit_cleanup(code=20, file=rsync.c, line=225): entered
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225)
_exit_cleanup(code=20, file=rsync.c, line=225): about to call exit(20)

Here is the output of rsync via ssh:

server2 # /tmp/rsync -avvvvv -e ssh --rsync-path=/tmp/rsync --progress 
--stats file1.db server1:/spare
cmd=ssh machine=server1 user= path=/spare
cmd=ssh server1 /tmp/rsync --server -vvvvvlogDtpr . /spare
opening connection using ssh server1 /tmp/rsync --server -vvvvvlogDtpr . 
/spare
Password:
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/3
Environment:
  USER=root
  LOGNAME=root
  HOME=/root
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/opt/ssh/bin
  MAIL=/var/mail/root
  SHELL=/sbin/sh
  TZ=EST5EDT
  SSH_CLIENT=192.168.12.39 58685 22
  SSH_CONNECTION=192.168.12.39 58685 192.168.15.232 22
(Client) Protocol versions: remote=28, negotiated=28
building file list ...
[sender] make_file(file1.db,*,2)
[sender] expand file_list to 131072 bytes, did move
1 file to consider
[sender] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 gid=3
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=24873
(Server) Protocol versions: remote=28, negotiated=28
recv_file_name(file1.db)
received 1 names
gid 3(sys) maps to 3
[receiver] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 
gid=3
recv_file_list done
get_local_name count=1 /spare
recv_files(1) starting
generator starting pid=24873 count=1
delta transmission enabled
recv_generator(file1.db,0)
send_files(0, file1.db)
count=0 n=0 rem=0
send_files mapped file1.db of size 4304863232
calling match_sums file1.db
file1.db
2146861056  49%    1.26MB/s    0:27:49
Killed by signal 2. 1.26MB/s    0:27:49
_exit_cleanup(code=20, file=rsync.c, line=225): entered
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225)
_exit_cleanup(code=20, file=rsync.c, line=225): about to call exit(20)

2146861056  49%    1.26MB/s    0:27:49

Here is the output for a local filesystem:

server2 # /tmp/rsync -avvvvv  --progress --stats file1.db 
/var/opt/ignite/test
cmd= machine= user= path=/var/opt/ignite/test
cmd=. /var/opt/ignite/test
(Client) Protocol versions: remote=28, negotiated=28
(Server) Protocol versions: remote=28, negotiated=28
building file list ...
[sender] make_file(file1.db,*,2)
[sender] expand file_list to 131072 bytes, did move
1 file to consider
[sender] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 gid=3
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=17592
recv_file_name(file1.db)
received 1 names
gid 3(sys) maps to 3
[receiver] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 
gid=3
recv_file_list done
get_local_name count=1 /var/opt/ignite/test
recv_files(1) starting
generator starting pid=17592 count=1
delta-transmission disabled for local transfer or --whole-file
recv_generator(file1.db,0)
send_files(0, file1.db)
count=0 n=0 rem=0
send_files mapped file1.db of size 4304863232
calling match_sums file1.db
file1.db
2142535680  49%    4.27MB/s    0:08:14
_exit_cleanup(code=20, file=rsync.c, line=225): entered
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225)
_exit_cleanup(code=20, file=rsync.c, line=225): about to call exit(20)
rsync: writefd_unbuffered failed to write 60 bytes: phase "unknown": Broken 
pipe
_exit_cleanup(code=12, file=io.c, line=836): entered
rsync error: error in rsync protocol data stream (code 12) at io.c(836)
_exit_cleanup(code=12, file=io.c, line=836): about to call exit(12)


--------------------------------------------------------------------------------


HP-UX 11i and largefiles on rsync 2.6.2
Don Malloy nixuser23 at hotmail.com
Thu Jul 15 20:30:50 GMT 2004

--------------------------------------------------------------------------------

Interestingly enough, now it stops dead when it hits 2146631680 bytes
transferred.

I've attached the output as a file.

The rsync from /tmp is the patched one (I modified generator.c as noted and
recompiled)

Any ideas?

Thanks again,
Don

>From: Wayne Davison <wayned at samba.org>
>To: Don Malloy <nixuser23 at hotmail.com>
>CC: rsync at lists.samba.org
>Subject: Re: HP-UX 11i and largefiles on rsync 2.6.2
>Date: Thu, 15 Jul 2004 09:28:10 -0700
>
>On Wed, Jul 14, 2004 at 04:46:41PM -0400, Don Malloy wrote:
> > The rsync transfers always fail at the 2GB mark.
>
>There is a bug in the large-file support of 2.6.2 that is fixed by the
>attached patch.  Either apply it using "patch" or manually edit the
>size_t to OFF_T manually, recompile, and give the resulting rsync a try.
>
>..wayne..
><< large-files.patch >>

_________________________________________________________________
STOP MORE SPAM with the MSN Premium and get 2 months FREE*
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
-------------- next part --------------
Patched version:

server2 # /tmp/rsync -avvvv -e remsh --rsync-path=/tmp/rsync --progress
--stats file1.db server1:/spare
opening connection using remsh server1 /tmp/rsync --server -vvvlogDtpr .
/spare
cmd=remsh machine=server1 user= path=/spare
cmd=remsh server1 /tmp/rsync --server -vvvvlogDtpr . /spare
opening connection using remsh server1 /tmp/rsync --server -vvvvlogDtpr .
/spare
(Client) Protocol versions: remote=28, negotiated=28
(Server) Protocol versions: remote=28, negotiated=28
building file list ...
[sender] make_file(file1.db,*,2)
[sender] expand file_list to 131072 bytes, did move
1 file to consider
[sender] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0 gid=3
send_file_list done
file list sent
send_files starting
server_recv(2) starting pid=10383
recv_file_name(file1.db)
received 1 names
gid 3(sys) maps to 3
[receiver] i=0 <NULL> <NULL> file1.db mode=0100640 len=4304863232 uid=0
gid=3
recv_file_list done
get_local_name count=1 /spare
recv_files(1) starting
generator starting pid=10383 count=1
delta transmission enabled
recv_generator(file1.db,0)
send_files(0, file1.db)
count=0 n=0 rem=0
send_files mapped file1.db of size 4304863232
calling match_sums file1.db
file1.db
2146631680  49%    3.07MB/s    0:11:26 <-- hangs at this point - CTRL-c
_exit_cleanup(code=20, file=rsync.c, line=225): entered
rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225)
_exit_cleanup(code=20, file=rsync.c, line=225): about to call exit(20)
server2 #



Unpatched version:

server2 # /usr/local/bin/rsync -avvv -e remsh
--rsync-path=/usr/local/bin/rsync --progress --stats file1.db server1:/spare
opening connection using remsh server1 /usr/local/bin/rsync --server
-vvvlogDtpr . /spare
building file list ...
[sender] make_file(file1.db,*,2)
[sender] expand file_list to 131072 bytes, did move
1 file to consider
send_file_list done
send_files starting
server_recv(2) starting pid=10278
recv_file_name(file1.db)
received 1 names
recv_file_list done
get_local_name count=1 /spare
recv_files(1) starting
generator starting pid=10278 count=1
delta transmission enabled
recv_generator(file1.db,0)
send_files(0, file1.db)
send_files mapped file1.db of size 4304863232
calling match_sums file1.db
file1.db
2127200256  49%    3.17MB/s    0:11:10
rsync: writefd_unbuffered failed to write 32768 bytes: phase "unknown":
Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(836)
_exit_cleanup(code=12, file=io.c, line=836): about to call exit(12)

_________________________________________________________________
MSN Premium helps eliminate e-mail viruses. Get 2 months FREE*  
http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines



More information about the rsync mailing list