Fwd: rsync ingest to new storage environment

Herb Burnswell herbert.burnswell at gmail.com
Thu Oct 19 03:49:48 UTC 2017


> If rsync isn't doing the networking you are better off with cp -au
> instead of rsync.  It should be significantly faster and you can do a
> final pass with rsync to get any files that got truncated by a ^C (cp
> can only skip files that are newer not files that are not different and
> a truncated file will be newer since it never got back-dated).

Thanks, I will run some tests.  Is there any performance increase from
allowing rsync doing the networking?

>>      rsync -av --no-perms --no-owner --no-group /old/storage/dir
>> /new/storage/directory/dir

> Note that rsync treats a trailing / on the source parameter differently.
> If you did rsync ... /old/storage/dir /new/storage/dir then you made
> /new/storage/dir/dir and duplicated everything into it.  The correct
> syntax is rsync ... /old/storage/dir/ /new/storage/dir OR rsync ...
> /old/storage/dir /new/storage

Yes, I just typed it wrong in the example..




On 10/18/2017 05:00 PM, Herb Burnswell via rsync wrote:
> All,
>
> I am seeding a new storage environment (Glusterfs on XFS) and would like
> to gather advise on best practices.  This data is primarily all media
> data, so not good with compression.
>
> I currently have made one pass on at 20TB directory tree into the
> environment as:
>
> - nfs mount from old storage to new storage
> - rsync -av /old/storage/* /new/storage/directory
>
> Once the directories and files were on the new storage, I did:
>
> - chown -R root:root
> - chmod -R 774
>
> I'll need to do a couple more sync's prior to full cut over.
>
> Questions regarding performance:
>
> - Does anyone have any suggestions on how to achieve the best performace
> (speed)?
>
>      - Is a local NFS mount from old storage to new storage the best
> option?  If so are there specific mount options that should be used?
>      - Any specific rsync flags (I've tested with and without 'z' flag
> and it does not help with this data) or best practices?
>
> Questions regarding rsync behavior:
>
> - When I test individual directory resync's within the initial ingest
> tree, a command such as:
>
>      rsync -av --no-perms --no-owner --no-group /old/storage/dir
> /new/storage/directory/dir
>
>  Lists all of the directories under 'dir' in the shell.  But if I rerun
> the command immediately thereafter, nothing is listed in the shell.
> Where is this 'metadata' of what is 'already on the destination'
> stored?  Is it only stored while the shell is open?  I want to set up a
> cron job moving forward and would like to make sure all info is available.
>
> Any guidance is greatly appreciated.
>
> Thanks in advance,
>
> HB
>
>
>
>

--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
        Orlando, Florida                kmk at sanitarium.net (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/
mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20171018/d188a359/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 231 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/rsync/attachments/20171018/d188a359/signature.sig>


More information about the rsync mailing list