What is it doing?

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It is still the same number of stat calls.  Doesn't really matter if
you split them up.

Can you rsync to the NFS server directly?

On 01/13/2014 05:34 PM, Perry Smith wrote:
> Ok.  I can get the Mac up to version 3 but I'm wondering if I need
> to rethink my whole strategy.  Since the source is on NFS, doing a
> stat on all the files each run may cost me too much time.
> 
> I might need to split it into smaller pieces and then rotate 
> through the pieces via a script.  Do you have any suggestions for
> this type of situation?
> 
> Perry
> 
> On Jan 13, 2014, at 4:08 PM, Kevin Korb <kmk at sanitarium.net>
> wrote:
> 
>> Signed PGP part On 01/13/2014 05:05 PM, Perry Smith wrote:
>>> A friend and I noticed the --times or --archive flag.  I have
>>> not stopped it yet but I'll add that flag (probably --times).
>>> 
>>> This is the first time so it must be #2.
>>> 
>>> The side issuing the command is a Mac using rsync version
>>> 2.6.9 protocol version 29.  The other side is AIX using rsync
>>> version 3.1.0 protocol version 31 (that I built myself).
>> 
>> Yes, if either end is version 2 then rsync will have to index
>> the entire tree on both systems before it starts copying
>> anything.
>> 
>>> I don't mind recompiling rsync on the Mac side if you think
>>> that would improve things.
>> 
>> I have no Mac experience but that is the way it is everywhere
>> else.
>> 
>>> I was trying to find some type of scratch file or something
>>> but could not.  I'm curious, where is the index kept?
>> 
>> There is no index kept.  Rsync has no memory between runs which
>> is why copying the timestamps is important.
>> 
>> When I say indexing files I really mean it is going through the
>> tree and doing a stat() on everything so it will have a list of
>> existing files and timestamps to compare with the other end.
>> Rsync v3 does this too but it does it incrementally while it is
>> also copying stuff.
>> 
>>> Thank you for your help Perry
>>> 
>>> On Jan 13, 2014, at 2:49 PM, Kevin Korb <kmk at sanitarium.net> 
>>> wrote:
>>> 
>>>> Signed PGP part First, don't run rsync without either --times
>>>> or --archive.  Without that rsync won't copy timestamps and
>>>> it won't be able to tell what is changed when you run it
>>>> again.
>>>> 
>>>> Second, if rsync isn't copying anything then there are 2 
>>>> reasons... 1. You already have most of the files copied and
>>>> it is going through them looking for a file that needs
>>>> updating 2. You are using rsync version 2 where all files had
>>>> to be indexed before it copied anything.
>>>> 
>>>> On 01/13/2014 03:06 PM, Perry Smith wrote:
>>>>> This is my first time to really use rsync.  I did small
>>>>> tests to get the arguments like I wanted and then kicked
>>>>> off the big rsync about 2 and a half hours ago.  So far, it
>>>>> has not copied over any files.
>>>>> 
>>>>> The command I used is:
>>>>> 
>>>>> rsync \ --relative \ --recursive \ --copy-links \ 
>>>>> host:/glob/that/matches/about/eighty/./directories \ 
>>>>> /local/target/dir
>>>>> 
>>>>> The list of directories are all full of symbolic links
>>>>> that point off to NFS mounted file systems.  I don't expect
>>>>> it to complete today but I do have to stop it each day at
>>>>> the end of the work day. But it worries me that it has yet
>>>>> to copy over any files.
>>>>> 
>>>>> Is it really making progress?  Or will it take this long
>>>>> to really start copying files over each day I start it?
>>>>> 
>>>>> I expect the total amount copied to be about 400G and about
>>>>> 4 million files.
>>>>> 
>>>>> It is possible to break this up into pieces if that would 
>>>>> help.
>>>>> 
>>>>> Thank you for your help and advice, Perry
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> -- 
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>
>>>>
>>
>>>> 
Kevin Korb			Phone:    (407) 252-6853
>>>> Systems Administrator		Internet: FutureQuest, Inc. 
>>>> Kevin at FutureQuest.net  (work) Orlando, Florida 
>>>> kmk at sanitarium.net (personal) Web page: 
>>>> http://www.sanitarium.net/ PGP public key available on web
>>>> site. 
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>
>>>>
>>>>
>>
>>>> 
- --
>>>> Please use reply-all for most replies to avoid omitting the 
>>>> mailing list. To unsubscribe or change options: 
>>>> https://lists.samba.org/mailman/listinfo/rsync Before
>>>> posting, read:
>>>> http://www.catb.org/~esr/faqs/smart-questions.html
>>> 
>> 
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>> 
Kevin Korb			Phone:    (407) 252-6853
>> Systems Administrator		Internet: FutureQuest, Inc.
>> Kevin at FutureQuest.net  (work) Orlando, Florida
>> kmk at sanitarium.net (personal) Web page:
>> http://www.sanitarium.net/ PGP public key available on web site. 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>
>> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlLUbFcACgkQVKC1jlbQAQd6VwCfe8wnRBnfp02Bs5yq93LAJgNF
ASIAn0PWLh1yvfn49j6Pm4vR29eJCo0G
=ezKS
-----END PGP SIGNATURE-----