rsync of big image files from virtual hosts takes lot of time
Kevin Korb
kmk at sanitarium.net
Sat Oct 20 16:56:12 UTC 2018
First, add --stats to find out how much is being transferred. Second,
if these are fixed size or sparse image files then --sparse will be a
big help. Finally, if they are not fixed size or sparse then --inplace
is probably making things worse.
--no-whole-file is the default btw unless you are doing a local copy.
If you are doing a local copy then just use cp as it is much faster at that.
On 10/20/2018 12:36 PM, Lentes, Bernd via rsync wrote:
> Hi,
>
> i'd like to rsync some image files from virtual hosts each night. The image files are between 50GB and 400GB big.
> I thought rsync would be the appropriate solution because it just transfers the differences, not the whole file.
> But nevertheless rsync takes hours.
> The images aren't much modified over the day, i would expect just some tens of megabyte needed to be transfered.
>
> I have the following command line:
> rsync -av --no-whole-file --inplace source destination.
>
> Is that ok ? Any suggestions ?
>
> I skipped the check with the hash values before the transfer because creating a hash value from a 400GB file takes loooong.
> But the man pages mentione that nevertheless AFTER the transfer a hash value is created on both sides to check that the files are equal.
> Is it possible to switch the after-transfer check off ?
>
> I have SLES 12 SP3 and rsync 3.1.0
>
> Thanks.
>
> Bernd
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20181020/5f5a536b/signature.sig>
More information about the rsync
mailing list