help troubleshooting inconsistencies in back up sizes

[gaw zay]

On Fri, 4 Feb 2005, John Van Essen wrote:

> On Thu, 3 Feb 2005, gaw zay <gauze at dropdead.org> wrote:
> ....
> > I use this shell script to back up:
> >
> > for i in a b c d e f g h i j k l m n o p q r s t u v w x y z `seq 0 9`; do
> >         /usr/local/bin/rsync -a -z -W --delete /mailhome/$i/ user at backup:/mailhome/$i
> > done
> >
> > question one would be does this look correct?
>
> Your script looks OK, but it helps to use at least one -v option
> so you can see what files are being transferred.  If you have a busy
> mailserver at the time of the rsync, you might be needlessly
> transferring lock files and temporary files as mailboxes are updated.

doing maildir so some tmp files ,no locks ... I'll put some -v and --stats
on a job I am running on a smaller set of files (34G or so) later in the
afternoon to see what comes up.

> --stats would help you see the size of the hierarchies, too.
>
> You can remove -v and --stats once you get an idea of the activity.
>
> > now here is the heart of the problem:
> > on server1 the partition is 121G
> > on server2 it's 110G
> > on server3 it's 118G
>
> When you say the partition is NNN, do you mean that's the used space
> as reported by df?

yes. using df -h on same linux distro

>
> > so I assume I have multiple problems here. I don't see --progress as being
> > usable in my case since I have such a large amount of files, how can I
> > debug what the differences are between these 3 body of files that doesn't
> > involve actually checking them individually? I basically want to be
> > informed of errors of any kind (io, permissions) as my logging of
> > stderr/stdout doesn't show anything, which makes me think many the problem
> > is my command line but it sure looks ok to me. I'm using version 2.6.3 of
> > rsync btw.
>
> If you don't have -v or --stats or --progress then there will be
> no output for a successful rsync.
>
> You should be getting any error messages that may be generated.

ok good to know ...

>
> You could check the exit code from rsync and print your own message
> if it is non-zero to ensure that you get a message when there's
> an error.


yeah I should be doing this anyway but I don't, thinking rsync would spew
on error ....


>
> To examine the difference between hierarchies, do this on each server
> (replacing N with a number):
>
>   find /mailhome -ls | sort +10 >/tmp/serverN.txt
>
> (The sort sorts on the filename at the end of the line.)
>
> If the machines use different timezones, put TZ=GMT before the find:
>
>   TZ=GMT find /mailhome -ls | sort +10 >/tmp/serverN.txt
>
> so all the timestamps will be the same.
>
> Then use rsync to gather them on one machine and do diffs between them.
> --
>         John Van Essen  Univ of Minn. Alumnus  <vanes002 at umn.edu>
>


I was thinking of doing this and then something like:
cat server2.txt server3.txt | sort | uniq -u
to only get files that don't exist on both. of course it'll take 5000
years to run on 25 million lines, ugh.


Thanks for some ideas.
brian
--
Never be afraid to tell the world who you are.
             -- Anonymous
 07:30:01 up 4 days, 17:53, 10 users,  load average: 0.31, 0.32, 0.25