Very slow to start sync with millions of directories and files

[Bráulio Bhavamitra]

On Tue, Mar 20, 2018 at 5:49 PM Kevin Korb <kmk at sanitarium.net> wrote:

> Nothing there should be preventing incremental indexing.  That means it
> should start copying as soon as it finds a file that needs to be copied.
>
Doesn't it tries to create all (empty) directories first?


> On 03/20/2018 02:33 PM, Bráulio Bhavamitra wrote:
> >
> >
> > Em seg, 19 de mar de 2018 11:34, Kevin Korb via rsync
> > <rsync at lists.samba.org <mailto:rsync at lists.samba.org>> escreveu:
> >
> >     The performance of rsync with a huge number of files is greatly
> >     determined by every option you are using.  So, what is your whole
> >     command line?
> >
> > rsync -avP /data-old/ /data
> >
> >
> >     On 03/19/2018 09:05 AM, Bráulio Bhavamitra via rsync wrote:
> >     > Hi all,
> >     >
> >     > I'm using rsync 3 to copy all files from one disk to another. The
> >     files
> >     > were writen by Minio, an S3 compatible opensource backend.
> >     >
> >     > The number of files is dozens of millions, almost each of them
> within
> >     > its own directory.
> >     >
> >     > Rsync takes a long time, when not several hours, to even start
> syncing
> >     > files. I already see a few reasons:
> >     > - it first create all directories to put files in, that could be
> done
> >     > along with the sync
> >     > - it needs to generate the list of all files before starting, and
> >     cannot
> >     > start syncing and keep the list generation in a different thread.
> >     >
> >     > Cheers,
> >     > bráulio
> >     >
> >     >
> >
> >     --
> >
>  ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >             Kevin Korb                      Phone:    (407) 252-6853
> >             Systems Administrator           Internet:
> >             FutureQuest, Inc.               Kevin at FutureQuest.net
> (work)
> >             Orlando, Florida                kmk at sanitarium.net
> >     <mailto:kmk at sanitarium.net> (personal)
> >             Web page:                       http://www.sanitarium.net/
> >             PGP public key available on web site.
> >
>  ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >
> >     --
> >     Please use reply-all for most replies to avoid omitting the mailing
> >     list.
> >     To unsubscribe or change options:
> >     https://lists.samba.org/mailman/listinfo/rsync
> >     Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
> >
>
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>         Kevin Korb                      Phone:    (407) 252-6853
>         Systems Administrator           Internet:
>         FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>         Orlando, Florida                kmk at sanitarium.net (personal)
>         Web page:                       http://www.sanitarium.net/
>         PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20180320/436822e8/attachment.html>