What is it doing?
Brian K. White
brian at aljex.com
Mon Jan 13 17:15:15 MST 2014
On 1/13/2014 6:19 PM, Perry Smith wrote:
> Yea. Shouldn't be hard to split up. The hard part is some type
> of dependable rotation.
>
> You mention "pause"... I have to disconnect so I assume that would
> "abort" the transfer. But that triggered another question: would
> daemon mode help in this situation? (I assume not. The daemon
> probably folks and the child does all the work and dies when the
> connection is lost.)
OK the Mac is a laptop that disconnects?
Yes in that case merely using version 3+ will not help enough.
And now it sounds like the nfs source is not only remote but probably
accessed via vpn too? And you can't get either a shell or an rsync
client or an rsync server on it?
In that case yes I guess you will have to break up the total job into
pieces yourself.
I'm not sure but you might also be getting little or no benefit from
rsync in this case. You are not using -c, so I *THINK* that means it is
only looking at timestamps, sizes, onership, and perms to decide what
files to send or skip, all of which it gets from one stat() per file.
And I *THINK* that means it's not reading the entire contents of every
file over nfs to your laptop, only the ones that it is going to transfer.
So I guess you are getting some benefit. Assuming I'm right, files that
will be skipped based on stat metadata, you only did a stat over nfs
over vpn and wan, and the remote rsync did one and the two rsyncs
compared those over your wan.
For files that will be sent, you are reading the entire file (no deltas)
from nfs to your laptop every time, then transmitting a delta to the
remote rsync.
I am guessing that there is no way to mount the nfs share from the
destination machine and just doing a plain copy? Probably need vpn
client setup which only exists on the laptop, or only works while you
are physically at work?
Of course remember that the initial copy will be zero benefit from rsync
no matter what. So the first run may be the only one that you have to
worry about breaking up.
--
bkw
> Perry
>
> On Jan 13, 2014, at 4:55 PM, Kevin Korb <kmk at sanitarium.net> wrote:
>
>> Signed PGP part
>> If you have to abort it then I suppose that makes sense. Otherwise
>> you could throttle or pause it.
>>
>> If you do have to split it up then it shouldn't be difficult. Your
>> original command was specifying multiple sources using a glob of some
>> kind so you would just need to alter that.
>>
>> On 01/13/2014 05:51 PM, Perry Smith wrote:
>>> The NFS server is off somewhere else, locked down. secure, blah
>>> blah.
>>>
>>> Doing it via a script that rotates is the same number of stat calls
>>> but it would start at a different place each day.
>>>
>>> If I start it day 1 and it gets 25% through the stat calls, on day
>>> 2, will rsync start where it left off or start back at the
>>> beginning? I figure since it does not save context, I would start
>>> back at the beginning.
>>>
>>> So if I rotate, it would start at a different point.
>>>
>>> On Jan 13, 2014, at 4:44 PM, Kevin Korb <kmk at sanitarium.net>
>>> wrote:
>>>
>>>> Signed PGP part It is still the same number of stat calls.
>>>> Doesn't really matter if you split them up.
>>>>
>>>> Can you rsync to the NFS server directly?
>>>>
>>>> On 01/13/2014 05:34 PM, Perry Smith wrote:
>>>>> Ok. I can get the Mac up to version 3 but I'm wondering if I
>>>>> need to rethink my whole strategy. Since the source is on NFS,
>>>>> doing a stat on all the files each run may cost me too much
>>>>> time.
>>>>>
>>>>> I might need to split it into smaller pieces and then rotate
>>>>> through the pieces via a script. Do you have any suggestions
>>>>> for this type of situation?
>>>>>
>>>>> Perry
>>>>>
>>>>> On Jan 13, 2014, at 4:08 PM, Kevin Korb <kmk at sanitarium.net>
>>>>> wrote:
>>>>>
>>>>>> Signed PGP part On 01/13/2014 05:05 PM, Perry Smith wrote:
>>>>>>> A friend and I noticed the --times or --archive flag. I
>>>>>>> have not stopped it yet but I'll add that flag (probably
>>>>>>> --times).
>>>>>>>
>>>>>>> This is the first time so it must be #2.
>>>>>>>
>>>>>>> The side issuing the command is a Mac using rsync version
>>>>>>> 2.6.9 protocol version 29. The other side is AIX using
>>>>>>> rsync version 3.1.0 protocol version 31 (that I built
>>>>>>> myself).
>>>>>>
>>>>>> Yes, if either end is version 2 then rsync will have to
>>>>>> index the entire tree on both systems before it starts
>>>>>> copying anything.
>>>>>>
>>>>>>> I don't mind recompiling rsync on the Mac side if you
>>>>>>> think that would improve things.
>>>>>>
>>>>>> I have no Mac experience but that is the way it is
>>>>>> everywhere else.
>>>>>>
>>>>>>> I was trying to find some type of scratch file or
>>>>>>> something but could not. I'm curious, where is the index
>>>>>>> kept?
>>>>>>
>>>>>> There is no index kept. Rsync has no memory between runs
>>>>>> which is why copying the timestamps is important.
>>>>>>
>>>>>> When I say indexing files I really mean it is going through
>>>>>> the tree and doing a stat() on everything so it will have a
>>>>>> list of existing files and timestamps to compare with the
>>>>>> other end. Rsync v3 does this too but it does it
>>>>>> incrementally while it is also copying stuff.
>>>>>>
>>>>>>> Thank you for your help Perry
>>>>>>>
>>>>>>> On Jan 13, 2014, at 2:49 PM, Kevin Korb
>>>>>>> <kmk at sanitarium.net> wrote:
>>>>>>>
>>>>>>>> Signed PGP part First, don't run rsync without either
>>>>>>>> --times or --archive. Without that rsync won't copy
>>>>>>>> timestamps and it won't be able to tell what is changed
>>>>>>>> when you run it again.
>>>>>>>>
>>>>>>>> Second, if rsync isn't copying anything then there are 2
>>>>>>>> reasons... 1. You already have most of the files copied
>>>>>>>> and it is going through them looking for a file that
>>>>>>>> needs updating 2. You are using rsync version 2 where all
>>>>>>>> files had to be indexed before it copied anything.
>>>>>>>>
>>>>>>>> On 01/13/2014 03:06 PM, Perry Smith wrote:
>>>>>>>>> This is my first time to really use rsync. I did
>>>>>>>>> small tests to get the arguments like I wanted and then
>>>>>>>>> kicked off the big rsync about 2 and a half hours ago.
>>>>>>>>> So far, it has not copied over any files.
>>>>>>>>>
>>>>>>>>> The command I used is:
>>>>>>>>>
>>>>>>>>> rsync \ --relative \ --recursive \ --copy-links \
>>>>>>>>> host:/glob/that/matches/about/eighty/./directories \
>>>>>>>>> /local/target/dir
>>>>>>>>>
>>>>>>>>> The list of directories are all full of symbolic links
>>>>>>>>> that point off to NFS mounted file systems. I don't
>>>>>>>>> expect it to complete today but I do have to stop it
>>>>>>>>> each day at the end of the work day. But it worries me
>>>>>>>>> that it has yet to copy over any files.
>>>>>>>>>
>>>>>>>>> Is it really making progress? Or will it take this
>>>>>>>>> long to really start copying files over each day I
>>>>>>>>> start it?
>>>>>>>>>
>>>>>>>>> I expect the total amount copied to be about 400G and
>>>>>>>>> about 4 million files.
>>>>>>>>>
>>>>>>>>> It is possible to break this up into pieces if that
>>>>>>>>> would help.
>>>>>>>>>
>>>>>>>>> Thank you for your help and advice, Perry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>> Kevin Korb Phone: (407) 252-6853
>>>>>>>> Systems Administrator Internet: FutureQuest, Inc.
>>>>>>>> Kevin at FutureQuest.net (work) Orlando, Florida
>>>>>>>> kmk at sanitarium.net (personal) Web page:
>>>>>>>> http://www.sanitarium.net/ PGP public key available on
>>>>>>>> web site.
>>>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>> --
>>>>>>>> Please use reply-all for most replies to avoid omitting
>>>>>>>> the mailing list. To unsubscribe or change options:
>>>>>>>> https://lists.samba.org/mailman/listinfo/rsync Before
>>>>>>>> posting, read:
>>>>>>>> http://www.catb.org/~esr/faqs/smart-questions.html
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>>>
>>>>>>
>>>>
>>>>>>
>> Kevin Korb Phone: (407) 252-6853
>>>>>> Systems Administrator Internet: FutureQuest, Inc.
>>>>>> Kevin at FutureQuest.net (work) Orlando, Florida
>>>>>> kmk at sanitarium.net (personal) Web page:
>>>>>> http://www.sanitarium.net/ PGP public key available on web
>>>>>> site.
>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>>>
>>>>>
>>>>>>
>>>>
>>>>>>
>> --
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>
>>>>
>> Kevin Korb Phone: (407) 252-6853
>>>> Systems Administrator Internet: FutureQuest, Inc.
>>>> Kevin at FutureQuest.net (work) Orlando, Florida
>>>> kmk at sanitarium.net (personal) Web page:
>>>> http://www.sanitarium.net/ PGP public key available on web site.
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>>>
>>>
>>>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. Kevin at FutureQuest.net (work)
>> Orlando, Florida kmk at sanitarium.net (personal)
>> Web page: http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>
>
>
More information about the rsync
mailing list