rsync creates wrong sized files
Dr A V Le Blanc
LeBlanc at mcc.ac.uk
Thu May 20 07:21:56 GMT 2004
I'm having a pretty serious rsync bug, which I've submitted to the
Debian bug system. But as the rsync maintainer there seems to be a bit
slow in fixing problems, I thought perhaps I should report it here
as well.
I'm using rsync 2.6.2 on a Debian woody system, with libc 2.2.5.
I have rsync running daily to mirror the Debian archives, mainly for
i386 files. The command I use is this:
rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
--exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
--exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
--exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
--exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
--exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
--exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
--exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
--exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
--exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
--exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
--exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
--exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
--exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
--password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
leblanc at ftp.uk.debian.org::debian/* /pub/linux/distributions/Debian
Every day this command leaves some wrong files in the tree; here is
an ls -l of /pub/linux/distributions/Debian/dists/sarge:
-rw-r--r-- 1 zlsiial root 6559910 May 5 20:22 Contents-alpha.gz
-rw-r--r-- 1 zlsiial sys 6353912 May 8 20:18 Contents-arm.gz
-rw-r--r-- 1 zlsiial sys 6322515 May 9 20:20 Contents-hppa.gz
-rw-r--r-- 1 zlsiial sys 7315705 May 9 20:23 Contents-i386.gz
-rw-r--r-- 1 zlsiial root 6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r-- 1 zlsiial root 6276314 May 2 20:18 Contents-m68k.gz
-rw-r--r-- 1 zlsiial root 6222678 May 2 20:20 Contents-mips.gz
-rw-r--r-- 1 zlsiial root 6247077 May 3 20:19 Contents-mipsel.gz
-rw-r--r-- 1 zlsiial root 6512485 May 3 20:22 Contents-powerpc.gz
-rw-r--r-- 1 zlsiial root 6328820 May 4 20:22 Contents-s390.gz
-rw-r--r-- 1 zlsiial root 6332818 May 4 20:25 Contents-sparc.gz
-rw-r--r-- 1 zlsiial root 6353912 May 12 20:23 Release
-rw-r--r-- 1 zlsiial root 6322515 May 12 20:23 Release.gpg
drwxr-xr-x 4 zlsiial root 2048 Jul 19 2002 contrib
drwxr-xr-x 6 zlsiial root 2048 May 3 13:38 main
Note that the Release and Release.gpg files are the same size as
Contents-arm.gz and Contents-hppa.gz respectively. I have checked,
and these files do _not_ have the same inode numbers, though their
contents are in fact the same. To correct this problem, I run
(currently manually) a different rsync job on the same machine:
rsync -r -R -l -t --delete --exclude=/Incoming --exclude=/UploadQueue \
--exclude=*_alpha.deb --exclude=*_alpha.udeb --exclude=*-alpha \
--exclude=*_arm.deb --exclude=*_arm.udeb --exclude=*-arm \
--exclude=*_hppa.deb --exclude=*_hppa.udeb --exclude=*-hppa \
--exclude=*_ia64.deb --exclude=*_ia64.udeb --exclude=*-ia64 \
--exclude=*_m68k.deb --exclude=*_m68k.udeb --exclude=*-m68k \
--exclude=*_mips.deb --exclude=*_mips.udeb --exclude=*-mips \
--exclude=*_mipsel.deb --exclude=*_mipsel.udeb --exclude=*-mipsel \
--exclude=*_powerpc.deb --exclude=*_powerpc.udeb --exclude=*-powerpc \
--exclude=*_s390.deb --exclude=*_s390.udeb --exclude=*-s390 \
--exclude=*_sparc.deb --exclude=*_sparc.udeb --exclude=*-sparc \
--exclude=*-alpha/* --exclude=*-arm/* --exclude=*-hppa/* --exclude=*-ia64/* \
--exclude=*-m68k/* --exclude=*-mips/* --exclude=*-mipsel/* \
--exclude=*-powerpc/* --exclude=*-s390/* --exclude=*-sparc/* \
--password-file /ftp/mirror/rsync.d/ftp.uk.debian.org \
leblanc at ftp.uk.debian.org::debian/dists/* /pub/linux/distributions/Debian
This is exactly the same as the first job with one exception: on the
last line '::debian/*' is changed to '::debian/dists/*'. After running
this job, the command 'ls -l /pub/linux/distributions/Debian/dists/sarge'
produces this output:
-rw-r--r-- 1 zlsiial root 6559910 May 5 20:22 Contents-alpha.gz
-rw-r--r-- 1 zlsiial sys 6353912 May 8 20:18 Contents-arm.gz
-rw-r--r-- 1 zlsiial sys 6322515 May 9 20:20 Contents-hppa.gz
-rw-r--r-- 1 zlsiial sys 7315705 May 9 20:23 Contents-i386.gz
-rw-r--r-- 1 zlsiial root 6409582 May 11 20:18 Contents-ia64.gz
-rw-r--r-- 1 zlsiial root 6276314 May 2 20:18 Contents-m68k.gz
-rw-r--r-- 1 zlsiial root 6222678 May 2 20:20 Contents-mips.gz
-rw-r--r-- 1 zlsiial root 6247077 May 3 20:19 Contents-mipsel.gz
-rw-r--r-- 1 zlsiial root 6512485 May 3 20:22 Contents-powerpc.gz
-rw-r--r-- 1 zlsiial root 6328820 May 4 20:22 Contents-s390.gz
-rw-r--r-- 1 zlsiial root 6332818 May 4 20:25 Contents-sparc.gz
-rw-r--r-- 1 zlsiial root 22828 May 12 20:23 Release
-rw-r--r-- 1 zlsiial root 315 May 12 20:23 Release.gpg
drwxr-xr-x 4 zlsiial root 2048 Jul 19 2002 contrib
drwxr-xr-x 6 zlsiial root 2048 May 3 13:38 main
Here you can see that the Release and Release.gpg files are the correct
sizes. I have noticed this problem almost daily in the dists directories,
but I have no idea whether there are files in the pool directories which
are the wrong size as well. All of our servers are running woody, and we
run rsync version 2.6.2-1, which I compiled from the (patched) source
using debuild.
A lot of the log of the faulty rsync run is taken up with error messages
like those below. The directories exist and are readable; for example,
ll /pub/linux/distributions/Debian/pool/main/m/mdcfg
total 228
-rw-r--r-- 1 zlsiial root 26262 May 5 04:02 mdcfg-utils_0.17_all.udeb
-rw-r--r-- 1 zlsiial root 35822 May 19 13:32 mdcfg-utils_0.18_all.udeb
-rw-r--r-- 1 zlsiial root 570 May 5 04:02 mdcfg_0.17.dsc
-rw-r--r-- 1 zlsiial root 76964 May 5 04:02 mdcfg_0.17.tar.gz
-rw-r--r-- 1 zlsiial root 3138 May 5 04:02 mdcfg_0.17_all.udeb
-rw-r--r-- 1 zlsiial root 570 May 19 13:32 mdcfg_0.18.dsc
-rw-r--r-- 1 zlsiial root 82425 May 19 13:32 mdcfg_0.18.tar.gz
-rw-r--r-- 1 zlsiial root 3182 May 19 13:32 mdcfg_0.18_all.udeb
I would be grateful for any help or advice with this problem.
-- Owen
LeBlanc at mcc.ac.uk
read errors mapping "pool/main/m/mdcfg" (in debian): (21) Is a directory
read errors mapping "pool/main/n/neon0.23" (in debian): (21) Is a directory
read errors mapping "pool/main/o/openggsn" (in debian): (21) Is a directory
read errors mapping "pool/main/s/snd" (in debian): (21) Is a directory
More information about the rsync
mailing list