help troubleshooting inconsistencies in back up sizes

gaw zay gauze at
Fri Feb 4 12:52:03 GMT 2005

On Fri, 4 Feb 2005, John Van Essen wrote:

> On Thu, 3 Feb 2005, gaw zay <gauze at> wrote:
> ....
> > I use this shell script to back up:
> >
> > for i in a b c d e f g h i j k l m n o p q r s t u v w x y z `seq 0 9`; do
> >         /usr/local/bin/rsync -a -z -W --delete /mailhome/$i/ user at backup:/mailhome/$i
> > done
> >
> > question one would be does this look correct?
> Your script looks OK, but it helps to use at least one -v option
> so you can see what files are being transferred.  If you have a busy
> mailserver at the time of the rsync, you might be needlessly
> transferring lock files and temporary files as mailboxes are updated.

doing maildir so some tmp files ,no locks ... I'll put some -v and --stats
on a job I am running on a smaller set of files (34G or so) later in the
afternoon to see what comes up.

> --stats would help you see the size of the hierarchies, too.
> You can remove -v and --stats once you get an idea of the activity.
> > now here is the heart of the problem:
> > on server1 the partition is 121G
> > on server2 it's 110G
> > on server3 it's 118G
> When you say the partition is NNN, do you mean that's the used space
> as reported by df?

yes. using df -h on same linux distro

> > so I assume I have multiple problems here. I don't see --progress as being
> > usable in my case since I have such a large amount of files, how can I
> > debug what the differences are between these 3 body of files that doesn't
> > involve actually checking them individually? I basically want to be
> > informed of errors of any kind (io, permissions) as my logging of
> > stderr/stdout doesn't show anything, which makes me think many the problem
> > is my command line but it sure looks ok to me. I'm using version 2.6.3 of
> > rsync btw.
> If you don't have -v or --stats or --progress then there will be
> no output for a successful rsync.
> You should be getting any error messages that may be generated.

ok good to know ...

> You could check the exit code from rsync and print your own message
> if it is non-zero to ensure that you get a message when there's
> an error.

yeah I should be doing this anyway but I don't, thinking rsync would spew
on error ....

> To examine the difference between hierarchies, do this on each server
> (replacing N with a number):
>   find /mailhome -ls | sort +10 >/tmp/serverN.txt
> (The sort sorts on the filename at the end of the line.)
> If the machines use different timezones, put TZ=GMT before the find:
>   TZ=GMT find /mailhome -ls | sort +10 >/tmp/serverN.txt
> so all the timestamps will be the same.
> Then use rsync to gather them on one machine and do diffs between them.
> --
>         John Van Essen  Univ of Minn. Alumnus  <vanes002 at>

I was thinking of doing this and then something like:
cat server2.txt server3.txt | sort | uniq -u
to only get files that don't exist on both. of course it'll take 5000
years to run on 25 million lines, ugh.

Thanks for some ideas.
Never be afraid to tell the world who you are.
             -- Anonymous
 07:30:01 up 4 days, 17:53, 10 users,  load average: 0.31, 0.32, 0.25

More information about the rsync mailing list