What is it doing?

Perry Smith pedzsan at gmail.com
Mon Jan 13 15:51:39 MST 2014


The NFS server is off somewhere else, locked down. secure, blah blah.

Doing it via a script that rotates is the same number of stat calls but it 
would start at a different place each day.

If I start it day 1 and it gets 25% through the stat calls, on day 2, will rsync
start where it left off or start back at the beginning?  I figure since it does
not save context, I would start back at the beginning.

So if I rotate, it would start at a different point.

On Jan 13, 2014, at 4:44 PM, Kevin Korb <kmk at sanitarium.net> wrote:

> Signed PGP part
> It is still the same number of stat calls.  Doesn't really matter if
> you split them up.
> 
> Can you rsync to the NFS server directly?
> 
> On 01/13/2014 05:34 PM, Perry Smith wrote:
> > Ok.  I can get the Mac up to version 3 but I'm wondering if I need
> > to rethink my whole strategy.  Since the source is on NFS, doing a
> > stat on all the files each run may cost me too much time.
> >
> > I might need to split it into smaller pieces and then rotate
> > through the pieces via a script.  Do you have any suggestions for
> > this type of situation?
> >
> > Perry
> >
> > On Jan 13, 2014, at 4:08 PM, Kevin Korb <kmk at sanitarium.net>
> > wrote:
> >
> >> Signed PGP part On 01/13/2014 05:05 PM, Perry Smith wrote:
> >>> A friend and I noticed the --times or --archive flag.  I have
> >>> not stopped it yet but I'll add that flag (probably --times).
> >>>
> >>> This is the first time so it must be #2.
> >>>
> >>> The side issuing the command is a Mac using rsync version
> >>> 2.6.9 protocol version 29.  The other side is AIX using rsync
> >>> version 3.1.0 protocol version 31 (that I built myself).
> >>
> >> Yes, if either end is version 2 then rsync will have to index
> >> the entire tree on both systems before it starts copying
> >> anything.
> >>
> >>> I don't mind recompiling rsync on the Mac side if you think
> >>> that would improve things.
> >>
> >> I have no Mac experience but that is the way it is everywhere
> >> else.
> >>
> >>> I was trying to find some type of scratch file or something
> >>> but could not.  I'm curious, where is the index kept?
> >>
> >> There is no index kept.  Rsync has no memory between runs which
> >> is why copying the timestamps is important.
> >>
> >> When I say indexing files I really mean it is going through the
> >> tree and doing a stat() on everything so it will have a list of
> >> existing files and timestamps to compare with the other end.
> >> Rsync v3 does this too but it does it incrementally while it is
> >> also copying stuff.
> >>
> >>> Thank you for your help Perry
> >>>
> >>> On Jan 13, 2014, at 2:49 PM, Kevin Korb <kmk at sanitarium.net>
> >>> wrote:
> >>>
> >>>> Signed PGP part First, don't run rsync without either --times
> >>>> or --archive.  Without that rsync won't copy timestamps and
> >>>> it won't be able to tell what is changed when you run it
> >>>> again.
> >>>>
> >>>> Second, if rsync isn't copying anything then there are 2
> >>>> reasons... 1. You already have most of the files copied and
> >>>> it is going through them looking for a file that needs
> >>>> updating 2. You are using rsync version 2 where all files had
> >>>> to be indexed before it copied anything.
> >>>>
> >>>> On 01/13/2014 03:06 PM, Perry Smith wrote:
> >>>>> This is my first time to really use rsync.  I did small
> >>>>> tests to get the arguments like I wanted and then kicked
> >>>>> off the big rsync about 2 and a half hours ago.  So far, it
> >>>>> has not copied over any files.
> >>>>>
> >>>>> The command I used is:
> >>>>>
> >>>>> rsync \ --relative \ --recursive \ --copy-links \
> >>>>> host:/glob/that/matches/about/eighty/./directories \
> >>>>> /local/target/dir
> >>>>>
> >>>>> The list of directories are all full of symbolic links
> >>>>> that point off to NFS mounted file systems.  I don't expect
> >>>>> it to complete today but I do have to stop it each day at
> >>>>> the end of the work day. But it worries me that it has yet
> >>>>> to copy over any files.
> >>>>>
> >>>>> Is it really making progress?  Or will it take this long
> >>>>> to really start copying files over each day I start it?
> >>>>>
> >>>>> I expect the total amount copied to be about 400G and about
> >>>>> 4 million files.
> >>>>>
> >>>>> It is possible to break this up into pieces if that would
> >>>>> help.
> >>>>>
> >>>>> Thank you for your help and advice, Perry
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> >>>>
> >>>>
> >>
> >>>>
> Kevin Korb			Phone:    (407) 252-6853
> >>>> Systems Administrator		Internet: FutureQuest, Inc.
> >>>> Kevin at FutureQuest.net  (work) Orlando, Florida
> >>>> kmk at sanitarium.net (personal) Web page:
> >>>> http://www.sanitarium.net/ PGP public key available on web
> >>>> site.
> >>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> >>>>
> >>>>
> >>>>
> >>
> >>>>
> --
> >>>> Please use reply-all for most replies to avoid omitting the
> >>>> mailing list. To unsubscribe or change options:
> >>>> https://lists.samba.org/mailman/listinfo/rsync Before
> >>>> posting, read:
> >>>> http://www.catb.org/~esr/faqs/smart-questions.html
> >>>
> >>
> >> --
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> >>
> >>
> Kevin Korb			Phone:    (407) 252-6853
> >> Systems Administrator		Internet: FutureQuest, Inc.
> >> Kevin at FutureQuest.net  (work) Orlando, Florida
> >> kmk at sanitarium.net (personal) Web page:
> >> http://www.sanitarium.net/ PGP public key available on web site.
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> >>
> >
> >>
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			http://www.sanitarium.net/
> 	PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.samba.org/pipermail/rsync/attachments/20140113/d6c8b1fb/attachment.pgp>


More information about the rsync mailing list