rsync creates wrong sized files

Dr A V Le Blanc LeBlanc at mcc.ac.uk
Thu May 20 07:21:56 GMT 2004


I'm having a pretty serious rsync bug, which I've submitted to the
Debian bug system.  But as the rsync maintainer there seems to be a bit
slow in fixing problems, I thought perhaps I should report it here
as well.

I'm using rsync 2.6.2 on a Debian woody system, with libc 2.2.5.
I have rsync running daily to mirror the Debian archives, mainly for
i386 files.  The command I use is this:

rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
 --exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
 --exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
 --exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
 --exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
 --exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
 --exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
 --exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
 --exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
 --exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
 --exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
 --exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
 --exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
 --exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
 --password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
 leblanc at ftp.uk.debian.org::debian/* /pub/linux/distributions/Debian

Every day this command leaves some wrong files in the tree; here is
an ls -l of /pub/linux/distributions/Debian/dists/sarge:

-rw-r--r--    1 zlsiial  root      6559910 May  5 20:22 Contents-alpha.gz
-rw-r--r--    1 zlsiial  sys       6353912 May  8 20:18 Contents-arm.gz
-rw-r--r--    1 zlsiial  sys       6322515 May  9 20:20 Contents-hppa.gz
-rw-r--r--    1 zlsiial  sys       7315705 May  9 20:23 Contents-i386.gz
-rw-r--r--    1 zlsiial  root      6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r--    1 zlsiial  root      6276314 May  2 20:18 Contents-m68k.gz
-rw-r--r--    1 zlsiial  root      6222678 May  2 20:20 Contents-mips.gz
-rw-r--r--    1 zlsiial  root      6247077 May  3 20:19 Contents-mipsel.gz
-rw-r--r--    1 zlsiial  root      6512485 May  3 20:22 Contents-powerpc.gz
-rw-r--r--    1 zlsiial  root      6328820 May  4 20:22 Contents-s390.gz
-rw-r--r--    1 zlsiial  root      6332818 May  4 20:25 Contents-sparc.gz
-rw-r--r--    1 zlsiial  root      6353912 May 12 20:23 Release
-rw-r--r--    1 zlsiial  root      6322515 May 12 20:23 Release.gpg
drwxr-xr-x    4 zlsiial  root         2048 Jul 19  2002 contrib
drwxr-xr-x    6 zlsiial  root         2048 May  3 13:38 main

Note that the Release and Release.gpg files are the same size as
Contents-arm.gz and Contents-hppa.gz respectively.  I have checked,
and these files do _not_ have the same inode numbers, though their
contents are in fact the same.  To correct this problem, I run
(currently manually) a different rsync job on the same machine:

rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
 --exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
 --exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
 --exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
 --exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
 --exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
 --exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
 --exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
 --exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
 --exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
 --exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
 --exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
 --exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
 --exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
 --password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
 leblanc at ftp.uk.debian.org::debian/dists/* /pub/linux/distributions/Debian

This is exactly the same as the first job with one exception: on the
last line '::debian/*' is changed to '::debian/dists/*'.  After running
this job, the command 'ls -l /pub/linux/distributions/Debian/dists/sarge'
produces this output:

-rw-r--r--    1 zlsiial  root      6559910 May  5 20:22 Contents-alpha.gz
-rw-r--r--    1 zlsiial  sys       6353912 May  8 20:18 Contents-arm.gz
-rw-r--r--    1 zlsiial  sys       6322515 May  9 20:20 Contents-hppa.gz
-rw-r--r--    1 zlsiial  sys       7315705 May  9 20:23 Contents-i386.gz
-rw-r--r--    1 zlsiial  root      6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r--    1 zlsiial  root      6276314 May  2 20:18 Contents-m68k.gz
-rw-r--r--    1 zlsiial  root      6222678 May  2 20:20 Contents-mips.gz
-rw-r--r--    1 zlsiial  root      6247077 May  3 20:19 Contents-mipsel.gz
-rw-r--r--    1 zlsiial  root      6512485 May  3 20:22 Contents-powerpc.gz
-rw-r--r--    1 zlsiial  root      6328820 May  4 20:22 Contents-s390.gz
-rw-r--r--    1 zlsiial  root      6332818 May  4 20:25 Contents-sparc.gz
-rw-r--r--    1 zlsiial  root        22828 May 12 20:23 Release
-rw-r--r--    1 zlsiial  root          315 May 12 20:23 Release.gpg
drwxr-xr-x    4 zlsiial  root         2048 Jul 19  2002 contrib
drwxr-xr-x    6 zlsiial  root         2048 May  3 13:38 main

Here you can see that the Release and Release.gpg files are the correct
sizes.  I have noticed this problem almost daily in the dists directories,
but I have no idea whether there are files in the pool directories which
are the wrong size as well.  All of our servers are running woody, and we
run rsync version 2.6.2-1, which I compiled from the (patched) source
using debuild.

A lot of the log of the faulty rsync run is taken up with error messages
like those below.  The directories exist and are readable; for example,

ll /pub/linux/distributions/Debian/pool/main/m/mdcfg
total 228
-rw-r--r--    1 zlsiial  root        26262 May  5 04:02 mdcfg-utils_0.17_all.udeb
-rw-r--r--    1 zlsiial  root        35822 May 19 13:32 mdcfg-utils_0.18_all.udeb
-rw-r--r--    1 zlsiial  root          570 May  5 04:02 mdcfg_0.17.dsc
-rw-r--r--    1 zlsiial  root        76964 May  5 04:02 mdcfg_0.17.tar.gz
-rw-r--r--    1 zlsiial  root         3138 May  5 04:02 mdcfg_0.17_all.udeb
-rw-r--r--    1 zlsiial  root          570 May 19 13:32 mdcfg_0.18.dsc
-rw-r--r--    1 zlsiial  root        82425 May 19 13:32 mdcfg_0.18.tar.gz
-rw-r--r--    1 zlsiial  root         3182 May 19 13:32 mdcfg_0.18_all.udeb

I would be grateful for any help or advice with this problem.

     -- Owen
     LeBlanc at mcc.ac.uk

read errors mapping "pool/main/m/mdcfg" (in debian): (21) Is a directory
read errors mapping "pool/main/n/neon0.23" (in debian): (21) Is a directory
read errors mapping "pool/main/o/openggsn" (in debian): (21) Is a directory
read errors mapping "pool/main/s/snd" (in debian): (21) Is a directory


More information about the rsync mailing list