rsync ingest to new storage environment

Herb Burnswell herbert.burnswell at gmail.com
Wed Oct 18 21:00:37 UTC 2017


All,

I am seeding a new storage environment (Glusterfs on XFS) and would like to
gather advise on best practices.  This data is primarily all media data, so
not good with compression.

I currently have made one pass on at 20TB directory tree into the
environment as:

- nfs mount from old storage to new storage
- rsync -av /old/storage/* /new/storage/directory

Once the directories and files were on the new storage, I did:

- chown -R root:root
- chmod -R 774

I'll need to do a couple more sync's prior to full cut over.

Questions regarding performance:

- Does anyone have any suggestions on how to achieve the best performace
(speed)?

     - Is a local NFS mount from old storage to new storage the best
option?  If so are there specific mount options that should be used?
     - Any specific rsync flags (I've tested with and without 'z' flag and
it does not help with this data) or best practices?

Questions regarding rsync behavior:

- When I test individual directory resync's within the initial ingest tree,
a command such as:

     rsync -av --no-perms --no-owner --no-group /old/storage/dir
/new/storage/directory/dir

 Lists all of the directories under 'dir' in the shell.  But if I rerun the
command immediately thereafter, nothing is listed in the shell.  Where is
this 'metadata' of what is 'already on the destination' stored?  Is it only
stored while the shell is open?  I want to set up a cron job moving forward
and would like to make sure all info is available.

Any guidance is greatly appreciated.

Thanks in advance,

HB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20171018/41dd5335/attachment.html>


More information about the rsync mailing list