Very slow to start sync with millions of directories and files

[Kevin Korb]

The performance of rsync with a huge number of files is greatly
determined by every option you are using.  So, what is your whole
command line?

On 03/19/2018 09:05 AM, Bráulio Bhavamitra via rsync wrote:
> Hi all,
>  
> I'm using rsync 3 to copy all files from one disk to another. The files
> were writen by Minio, an S3 compatible opensource backend.
> 
> The number of files is dozens of millions, almost each of them within
> its own directory.
> 
> Rsync takes a long time, when not several hours, to even start syncing
> files. I already see a few reasons:
> - it first create all directories to put files in, that could be done
> along with the sync
> - it needs to generate the list of all files before starting, and cannot
> start syncing and keep the list generation in a different thread.
> 
> Cheers,
> bráulio
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20180319/580933bf/signature.sig>